[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:00 dist-test-slave-cvwc krb5kdc[16260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:00 dist-test-slave-cvwc krb5kdc[16260](info): set up 2 sockets
Aug 09 03:30:00 dist-test-slave-cvwc krb5kdc[16260](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 03:30:01.738119 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:01 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:01.856635 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:34351
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:34351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:01.978006 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:01.978283 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:01.978327 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:01.982331 16280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:01.982388 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:01.982405 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:01.982424 16280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:01.982452 16280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:01.987993 16280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:34351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:01.989120 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:01.990233 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:01.999589 16293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:01.999702 16280 server_base.cc:1034] running on GCE node
W20260809 03:30:01.999648 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:01.999660 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:02.000437 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:02.001992 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:02.003160 16280 hybrid_clock.cc:648] HybridClock initialized: now 1786246202003142 us; error 72 us; skew 500 ppm
Aug 09 03:30:01 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246201, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:02.005975 16280 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:02.006873 16280 webserver.cc:460] Webserver started at http://127.15.222.126:38655/ using document root <none> and password file <none>
I20260809 03:30:02.007467 16280 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:02.007537 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:02.007862 16280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:02.009976 16280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "af0979a4dfb948faa6c829e3b47d30ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "afd0a40286acfd016fdab908c12513e1"
server_key_iv: "1a01ad78b9e550e7e00fc31a9bd4c378"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.010558 16280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "af0979a4dfb948faa6c829e3b47d30ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "afd0a40286acfd016fdab908c12513e1"
server_key_iv: "1a01ad78b9e550e7e00fc31a9bd4c378"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.014604 16280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 03:30:02.016824 16308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.018150 16280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:02.018275 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "af0979a4dfb948faa6c829e3b47d30ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "afd0a40286acfd016fdab908c12513e1"
server_key_iv: "1a01ad78b9e550e7e00fc31a9bd4c378"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.018378 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:02.043692 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:02.046663 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:02.046895 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:02.054492 16280 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:34351
I20260809 03:30:02.054505 16390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:34351 every 8 connection(s)
I20260809 03:30:02.055503 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:02.058454 16391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:02.062532 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16280
I20260809 03:30:02.062674 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:02.063941 16249 external_mini_cluster.cc:1428] Setting key 85fa8e28ac86d72b45f09322eb0f39cb
I20260809 03:30:02.064105 16391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac: Bootstrap starting.
I20260809 03:30:02.066470 16391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:02.067232 16391 log.cc:824] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:02.069571 16391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac: No bootstrap required, opened a new log
I20260809 03:30:02.074469 16391 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } }
I20260809 03:30:02.074724 16391 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:02.074783 16391 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0979a4dfb948faa6c829e3b47d30ac, State: Initialized, Role: FOLLOWER
I20260809 03:30:02.075268 16391 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } }
I20260809 03:30:02.075438 16391 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:02.075508 16391 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:02.075610 16391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:02.076762 16391 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } }
I20260809 03:30:02.077132 16391 leader_election.cc:302] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0979a4dfb948faa6c829e3b47d30ac; no voters:
I20260809 03:30:02.077515 16391 leader_election.cc:288] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:02.077641 16396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:02.077891 16396 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [term 1 LEADER]: Becoming Leader. State: Replica: af0979a4dfb948faa6c829e3b47d30ac, State: Running, Role: LEADER
I20260809 03:30:02.078202 16396 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } }
I20260809 03:30:02.078723 16391 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:02.080013 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0979a4dfb948faa6c829e3b47d30ac. Latest consensus state: current_term: 1 leader_uuid: "af0979a4dfb948faa6c829e3b47d30ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } } }
I20260809 03:30:02.080063 16397 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0979a4dfb948faa6c829e3b47d30ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0979a4dfb948faa6c829e3b47d30ac" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34351 } } }
I20260809 03:30:02.080178 16397 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:02.080132 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:02.080567 16409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:02.084044 16409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:02.089540 16409 catalog_manager.cc:1347] Generated new cluster ID: f5cafd467fd5493e9a085bcf5c1a21a4
I20260809 03:30:02.089649 16409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:02.093694 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.067110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.15.222.126:34351)
0809 03:30:02.067596 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.067615 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.068848 (+ 1233us) server_negotiation.cc:407] Connection header received
0809 03:30:02.074070 (+ 5222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.074088 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.074398 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.074666 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.077678 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.079235 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.080962 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.081656 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.081848 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.085596 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.085601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.085613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.085638 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.089182 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.089854 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.089860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.089865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.090004 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.090411 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.090414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.090416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.090984 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.091169 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.091962 (+ 793us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.092265 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":155,"threads_started":1}
I20260809 03:30:02.100826 16409 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:02.101918 16409 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:02.106388 16409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0979a4dfb948faa6c829e3b47d30ac: Generated new TSK 0
I20260809 03:30:02.107018 16409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:02.138216 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:02.247251 16429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:02.247480 16429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:02.247522 16429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:02.251413 16429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:02.251466 16429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:02.251534 16429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:02.256626 16429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:02.257567 16429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:02.258566 16429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:02.267716 16442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.267743 16445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.267781 16441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:02.267966 16429 server_base.cc:1034] running on GCE node
I20260809 03:30:02.268373 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:02.268976 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:02.270129 16429 hybrid_clock.cc:648] HybridClock initialized: now 1786246202270113 us; error 29 us; skew 500 ppm
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:02.276856 16429 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:02.277756 16429 webserver.cc:460] Webserver started at http://127.15.222.65:41763/ using document root <none> and password file <none>
I20260809 03:30:02.278306 16429 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:02.278369 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:02.278570 16429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:02.280678 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "d5e0ce65a5764e4b9cff9f1c8f30b6eb"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "dd8779b9ba018e54bb4fd4c38fdd0d72"
server_key_iv: "79a4f4643987907f197f8e73cfc8dbd4"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.281237 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "d5e0ce65a5764e4b9cff9f1c8f30b6eb"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "dd8779b9ba018e54bb4fd4c38fdd0d72"
server_key_iv: "79a4f4643987907f197f8e73cfc8dbd4"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.284870 16429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:02.287108 16453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.288239 16429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:02.288352 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "d5e0ce65a5764e4b9cff9f1c8f30b6eb"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "dd8779b9ba018e54bb4fd4c38fdd0d72"
server_key_iv: "79a4f4643987907f197f8e73cfc8dbd4"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.288444 16429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:02.307456 16429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:02.310072 16429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:02.310237 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:02.310779 16429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:02.311571 16429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:02.311617 16429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.311669 16429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:02.311698 16429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.320180 16429 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:33679
I20260809 03:30:02.320205 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:33679 every 8 connection(s)
I20260809 03:30:02.321018 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:02.324011 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16429
I20260809 03:30:02.324119 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:02.324407 16249 external_mini_cluster.cc:1428] Setting key f7ad5393902ba47e9165fee9a5f72758
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:02.334985 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.322773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47857 (local address 127.15.222.126:34351)
0809 03:30:02.322924 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.322929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.323777 (+ 848us) server_negotiation.cc:407] Connection header received
0809 03:30:02.325403 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.325406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.325452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.325520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.326974 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.327493 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.328693 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.329042 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.329139 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.331561 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.331565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.331568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.331596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.333058 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.333500 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.333503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.333505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.333591 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.333921 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.333924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.333926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.334150 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.334255 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.334687 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.334811 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:02.335670 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.323208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34351 (local address 127.15.222.65:47857)
0809 03:30:02.323638 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:02.323678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:02.325189 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:02.325702 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:02.325709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:02.326187 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:02.326827 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.326846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.327609 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.327612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.328574 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.328585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.329170 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.329173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.329299 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.329860 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:02.329875 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:02.331394 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:02.333160 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.333164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.333175 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.333407 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.333673 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.333675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.333676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.333824 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.334246 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:02.334250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:02.334456 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 03:30:02.334591 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 03:30:02.334850 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20260809 03:30:02.336786 16630 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34351
I20260809 03:30:02.337025 16630 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:02.337533 16630 heartbeater.cc:511] Master 127.15.222.126:34351 requested a full tablet report, sending...
I20260809 03:30:02.339147 16332 ts_manager.cc:194] Registered new tserver with Master: d5e0ce65a5764e4b9cff9f1c8f30b6eb (127.15.222.65:33679)
I20260809 03:30:02.340221 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:47857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:02.350788 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:02.478654 16639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:02.478880 16639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:02.478921 16639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:02.482858 16639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:02.482913 16639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:02.482970 16639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:02.487939 16639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:02.488823 16639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:02.489763 16639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:02.499094 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.499138 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.499137 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:02.499176 16639 server_base.cc:1034] running on GCE node
I20260809 03:30:02.499653 16639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:02.500231 16639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:02.501380 16639 hybrid_clock.cc:648] HybridClock initialized: now 1786246202501364 us; error 30 us; skew 500 ppm
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:02.503948 16639 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:02.504755 16639 webserver.cc:460] Webserver started at http://127.15.222.66:42709/ using document root <none> and password file <none>
I20260809 03:30:02.505257 16639 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:02.505316 16639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:02.505512 16639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:02.507534 16639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "61f5eb65dd11459b8338a7f19b204343"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "88281ce29ec6e9c6cd548f4b38376883"
server_key_iv: "5e49c7b9bbf17f039c38bd84d46d0de6"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.508133 16639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "61f5eb65dd11459b8338a7f19b204343"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "88281ce29ec6e9c6cd548f4b38376883"
server_key_iv: "5e49c7b9bbf17f039c38bd84d46d0de6"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.511817 16639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 03:30:02.514010 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.515100 16639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:02.515194 16639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "61f5eb65dd11459b8338a7f19b204343"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "88281ce29ec6e9c6cd548f4b38376883"
server_key_iv: "5e49c7b9bbf17f039c38bd84d46d0de6"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.515286 16639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:02.532928 16639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:02.535382 16639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:02.535552 16639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:02.536068 16639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:02.536811 16639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:02.536856 16639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.536900 16639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:02.536929 16639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.545192 16639 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:41829
I20260809 03:30:02.545214 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:41829 every 8 connection(s)
I20260809 03:30:02.546056 16639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:02.550175 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16639
I20260809 03:30:02.550280 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:02.550566 16249 external_mini_cluster.cc:1428] Setting key a20236c8b4ecc3ece77ea561121d42a9
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:02.559827 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.547905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:33837 (local address 127.15.222.126:34351)
0809 03:30:02.548047 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.548050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.548853 (+ 803us) server_negotiation.cc:407] Connection header received
0809 03:30:02.550157 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.550160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.550219 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.550302 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.551640 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.552232 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.553686 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.554019 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.554113 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.556481 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.556485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.556488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.556514 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.557987 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.558433 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.558437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.558439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.558544 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.558815 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.558818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.558819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.559038 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.559135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.559557 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.559667 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:30:02.560524 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.548254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34351 (local address 127.15.222.66:33837)
0809 03:30:02.548704 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:02.548745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:02.549937 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:02.550554 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:02.550563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:02.551036 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:02.551510 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.551524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.552362 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.552365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.553581 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.553593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.554144 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.554147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.554283 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.554809 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:02.554825 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:02.556317 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:02.558092 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.558096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.558107 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.558340 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.558620 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.558622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.558624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.558738 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.559122 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:02.559126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:02.559333 (+ 207us) client_negotiation.cc:769] Sending connection context
0809 03:30:02.559482 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 03:30:02.559711 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:02.561565 16852 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34351
I20260809 03:30:02.561848 16852 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:02.562356 16852 heartbeater.cc:511] Master 127.15.222.126:34351 requested a full tablet report, sending...
I20260809 03:30:02.563403 16332 ts_manager.cc:194] Registered new tserver with Master: 61f5eb65dd11459b8338a7f19b204343 (127.15.222.66:41829)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
I20260809 03:30:02.564029 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:33837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:02.576432 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:02.685828 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:02.686077 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:02.686137 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:02.690034 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:02.690092 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:02.690167 16859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:02.695412 16859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34351
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:02.696470 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:02.697423 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:02.706725 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.706758 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:02.706758 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:02.707052 16859 server_base.cc:1034] running on GCE node
I20260809 03:30:02.707427 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:02.708055 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:02.709223 16859 hybrid_clock.cc:648] HybridClock initialized: now 1786246202709206 us; error 28 us; skew 500 ppm
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:02.711788 16859 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:02.712602 16859 webserver.cc:460] Webserver started at http://127.15.222.67:42891/ using document root <none> and password file <none>
I20260809 03:30:02.713141 16859 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:02.713208 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:02.713408 16859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:02.715461 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "a42453b2d73d4b48afb6c95f025fd4ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "6dbda668fb191f23d4a99546e811db9a"
server_key_iv: "be98524225bd5d63fcb35e46801cd6cb"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.716005 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "a42453b2d73d4b48afb6c95f025fd4ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "6dbda668fb191f23d4a99546e811db9a"
server_key_iv: "be98524225bd5d63fcb35e46801cd6cb"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.719805 16859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:02.722167 16886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.723357 16859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:02.723469 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "a42453b2d73d4b48afb6c95f025fd4ac"
format_stamp: "Formatted at 2026-08-09 03:30:02 on dist-test-slave-cvwc"
server_key: "6dbda668fb191f23d4a99546e811db9a"
server_key_iv: "be98524225bd5d63fcb35e46801cd6cb"
server_key_version: "encryptionkey@0"
I20260809 03:30:02.723562 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:02.736449 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:02.739001 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:02.739161 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:02.739681 16859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:02.740448 16859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:02.740494 16859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.740542 16859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:02.740569 16859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:02.748927 16859 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:41819
I20260809 03:30:02.748946 17055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:41819 every 8 connection(s)
I20260809 03:30:02.749752 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:02.752044 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16859
I20260809 03:30:02.752133 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:02.752368 16249 external_mini_cluster.cc:1428] Setting key 47978c42d1333509fe83bf6cc23bf1b0
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:02.762311 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.751429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:56841 (local address 127.15.222.126:34351)
0809 03:30:02.751566 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.751569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.752415 (+ 846us) server_negotiation.cc:407] Connection header received
0809 03:30:02.753821 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.753825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.753881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.753952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.755052 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.755570 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.756677 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.756892 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.756980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.759257 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.759260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.759262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.759286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.760368 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.760830 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.760834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.760836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.760915 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.761216 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.761219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.761221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.761403 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.761489 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.762012 (+ 523us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.762143 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:30:02.763054 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.751776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34351 (local address 127.15.222.67:56841)
0809 03:30:02.752252 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:02.752302 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:02.753599 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:02.754115 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:02.754122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:02.754455 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:02.754913 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.754927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.755686 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.755689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.756557 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:02.756566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.757027 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.757031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:02.757188 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.757667 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:02.757682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:02.759097 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:02.760483 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.760489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.760503 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.760738 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.760996 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:02.760999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:02.761001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:02.761129 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:02.761503 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:02.761509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:02.761776 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 03:30:02.761951 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 03:30:02.762254 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20260809 03:30:02.764068 17057 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34351
I20260809 03:30:02.764287 17057 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:02.764760 17057 heartbeater.cc:511] Master 127.15.222.126:34351 requested a full tablet report, sending...
I20260809 03:30:02.765864 16332 ts_manager.cc:194] Registered new tserver with Master: a42453b2d73d4b48afb6c95f025fd4ac (127.15.222.67:41819)
I20260809 03:30:02.766563 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:56841
I20260809 03:30:02.766602 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:02.793076 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.784218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.15.222.65:33679)
0809 03:30:02.784535 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.784548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.784578 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:30:02.784635 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.784642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.784832 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.784961 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.785621 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.786459 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.787304 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.787682 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.787777 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.789944 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.789952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.789965 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.789991 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.791267 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.791683 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.791690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.791692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.791775 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.792028 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.792031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.792033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.792266 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.792471 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.792712 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.792901 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260809 03:30:02.794021 16494 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60482
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:02.819918 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:02.812394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.15.222.65:33679)
0809 03:30:02.812537 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:02.812543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:02.812657 (+ 114us) server_negotiation.cc:407] Connection header received
0809 03:30:02.812804 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:02.812808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:02.812852 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:02.812916 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:02.813555 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.814112 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.814897 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:02.815136 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:02.815222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:02.817281 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:02.817284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:02.817286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:02.817308 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:02.818649 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.818995 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.818998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.819000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.819086 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:02.819296 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:02.819298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:02.819299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:02.819453 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:02.819538 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:02.819710 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 03:30:02.819800 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:30:02.820796 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16429
I20260809 03:30:02.837955 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16639
I20260809 03:30:02.846621 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16859
I20260809 03:30:02.854029 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16280
2026-08-09T03:30:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2279 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[17084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[17084](info): set up 2 sockets
Aug 09 03:30:02 dist-test-slave-cvwc krb5kdc[17084](info): commencing operation
krb5kdc: starting...
W20260809 03:30:03.995827 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:04.114779 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:32811
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:32811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:04.225031 17104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:04.225281 17104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:04.225334 17104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:04.229265 17104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:04.229323 17104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:04.229349 17104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:04.229377 17104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:04.229403 17104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:04.234411 17104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:32811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:32811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:04.235363 17104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:04.236306 17104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:04.244194 17110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.244211 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.244212 17115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.244680 17104 server_base.cc:1034] running on GCE node
I20260809 03:30:04.245206 17104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:04.246253 17104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:04.247416 17104 hybrid_clock.cc:648] HybridClock initialized: now 1786246204247402 us; error 29 us; skew 500 ppm
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:04.249989 17104 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:04.250818 17104 webserver.cc:460] Webserver started at http://127.15.222.126:37993/ using document root <none> and password file <none>
I20260809 03:30:04.251308 17104 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:04.251363 17104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:04.251544 17104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:04.253569 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "61969d8923bc49d99e6b27ab6303b1d1"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "b215e5f681ab5c491fb5cc100ec8f032"
server_key_iv: "7db177659b0b4d8224f42bf5d2b21612"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.254153 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "61969d8923bc49d99e6b27ab6303b1d1"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "b215e5f681ab5c491fb5cc100ec8f032"
server_key_iv: "7db177659b0b4d8224f42bf5d2b21612"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.257647 17104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:04.259950 17124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.261093 17104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:04.261197 17104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "61969d8923bc49d99e6b27ab6303b1d1"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "b215e5f681ab5c491fb5cc100ec8f032"
server_key_iv: "7db177659b0b4d8224f42bf5d2b21612"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.261291 17104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:04.279554 17104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:04.282521 17104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:04.282694 17104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:04.289348 17104 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:32811
I20260809 03:30:04.289356 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:32811 every 8 connection(s)
I20260809 03:30:04.290288 17104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:04.290537 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17104
I20260809 03:30:04.290627 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:04.290915 16249 external_mini_cluster.cc:1428] Setting key 983fcfdcab817663359fe63a24e2da18
I20260809 03:30:04.293370 17210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:04.299695 17210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1: Bootstrap starting.
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:04.302088 17210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:04.302922 17210 log.cc:824] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:04.304791 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.292286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.15.222.126:32811)
0809 03:30:04.292716 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:04.292730 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:04.292795 (+ 65us) server_negotiation.cc:407] Connection header received
0809 03:30:04.293412 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:04.293429 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:04.293819 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:04.294089 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:04.294951 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.295928 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.297073 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.297559 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.297689 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.300387 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:04.300392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:04.300404 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:04.300430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:04.302027 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.302486 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.302491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.302495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.302604 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.302864 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.302866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.302867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.303413 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:04.303598 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:04.303846 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 03:30:04.304127 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260809 03:30:04.304816 17210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1: No bootstrap required, opened a new log
I20260809 03:30:04.308178 17210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } }
I20260809 03:30:04.308394 17210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:04.308451 17210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61969d8923bc49d99e6b27ab6303b1d1, State: Initialized, Role: FOLLOWER
I20260809 03:30:04.308871 17210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } }
I20260809 03:30:04.309016 17210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:04.309078 17210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:04.309170 17210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:04.310006 17210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } }
I20260809 03:30:04.310310 17210 leader_election.cc:302] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61969d8923bc49d99e6b27ab6303b1d1; no voters:
I20260809 03:30:04.310591 17210 leader_election.cc:288] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:04.310650 17215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:04.310858 17215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 61969d8923bc49d99e6b27ab6303b1d1, State: Running, Role: LEADER
I20260809 03:30:04.311203 17215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } }
I20260809 03:30:04.311683 17210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:04.312834 17217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61969d8923bc49d99e6b27ab6303b1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } } }
I20260809 03:30:04.312857 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61969d8923bc49d99e6b27ab6303b1d1. Latest consensus state: current_term: 1 leader_uuid: "61969d8923bc49d99e6b27ab6303b1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61969d8923bc49d99e6b27ab6303b1d1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 32811 } } }
I20260809 03:30:04.313119 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:04.313107 17217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:04.313921 17234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:04.316849 17234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:04.322628 17234 catalog_manager.cc:1347] Generated new cluster ID: 22ead1a6be7c4feb8b16d5d674d4202c
I20260809 03:30:04.322680 17234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:04.331609 17234 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:04.332410 17234 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:04.348996 17234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61969d8923bc49d99e6b27ab6303b1d1: Generated new TSK 0
I20260809 03:30:04.349659 17234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:04.385648 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:04.502102 17244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:04.502368 17244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:04.502414 17244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:04.506294 17244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:04.506348 17244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:04.506405 17244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:04.511471 17244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:04.512531 17244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:04.513579 17244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:04.523000 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.522991 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.523026 17244 server_base.cc:1034] running on GCE node
W20260809 03:30:04.522989 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.523628 17244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:04.524220 17244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:04.525369 17244 hybrid_clock.cc:648] HybridClock initialized: now 1786246204525352 us; error 28 us; skew 500 ppm
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:04.528007 17244 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:04.528787 17244 webserver.cc:460] Webserver started at http://127.15.222.65:38367/ using document root <none> and password file <none>
I20260809 03:30:04.529300 17244 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:04.529361 17244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:04.529554 17244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:04.531590 17244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "ed658b0201b844149b39455d713558ef"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "175bb23e9106138a0efe2a2d5683e9f1"
server_key_iv: "9c6fa1dfbc824a05f3b334693b3a51e3"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.532141 17244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "ed658b0201b844149b39455d713558ef"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "175bb23e9106138a0efe2a2d5683e9f1"
server_key_iv: "9c6fa1dfbc824a05f3b334693b3a51e3"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.535846 17244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:04.538094 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.539219 17244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:04.539321 17244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "ed658b0201b844149b39455d713558ef"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "175bb23e9106138a0efe2a2d5683e9f1"
server_key_iv: "9c6fa1dfbc824a05f3b334693b3a51e3"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.539412 17244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:04.553795 17244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:04.557117 17244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:04.557313 17244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:04.557967 17244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:04.558841 17244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:04.558908 17244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.558976 17244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:04.559005 17244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.568006 17244 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:37771
I20260809 03:30:04.568030 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:37771 every 8 connection(s)
I20260809 03:30:04.568924 17244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:04.573001 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17244
I20260809 03:30:04.573115 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:04.573405 16249 external_mini_cluster.cc:1428] Setting key 3d719814bb2c39a024d400077ca9c3db
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:04.582410 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.570718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:59829 (local address 127.15.222.126:32811)
0809 03:30:04.570879 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:04.570897 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:04.571729 (+ 832us) server_negotiation.cc:407] Connection header received
0809 03:30:04.572573 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:04.572576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:04.572624 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:04.572696 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:04.574149 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.574733 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.575838 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.576072 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.576159 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.578799 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:04.578803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:04.578806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:04.578833 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:04.580319 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.580802 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.580807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.580808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.580906 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.581194 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.581198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.581199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.581452 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:04.581558 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:04.582074 (+ 516us) server_negotiation.cc:299] Negotiation successful
0809 03:30:04.582226 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 03:30:04.583173 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.571125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:32811 (local address 127.15.222.65:59829)
0809 03:30:04.571576 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:04.571619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:04.572382 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:04.572925 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:04.572935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:04.573418 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:04.574039 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.574054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.574849 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.574851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.575723 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.575732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.576209 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.576213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.576368 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.576964 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:04.576984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:04.578632 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:04.580432 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:04.580437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:04.580450 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:04.580704 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:04.580987 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:04.580989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:04.580990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:04.581113 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:04.581540 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:04.581545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:04.581824 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 03:30:04.581992 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 03:30:04.582265 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:04.584420 17435 heartbeater.cc:347] Connected to a master server at 127.15.222.126:32811
I20260809 03:30:04.584651 17435 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:04.585158 17435 heartbeater.cc:511] Master 127.15.222.126:32811 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:04.586768 17149 ts_manager.cc:194] Registered new tserver with Master: ed658b0201b844149b39455d713558ef (127.15.222.65:37771)
I20260809 03:30:04.588269 17149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:59829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:04.600714 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:04.716192 17441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:04.716575 17441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:04.716650 17441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:04.720753 17441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:04.720809 17441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:04.720872 17441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:04.726084 17441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:04.727169 17441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:04.728185 17441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:04.737821 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.737821 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.737823 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.738150 17441 server_base.cc:1034] running on GCE node
I20260809 03:30:04.738601 17441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:04.739312 17441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:04.740492 17441 hybrid_clock.cc:648] HybridClock initialized: now 1786246204740467 us; error 35 us; skew 500 ppm
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:04.743573 17441 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:04.744478 17441 webserver.cc:460] Webserver started at http://127.15.222.66:41733/ using document root <none> and password file <none>
I20260809 03:30:04.745055 17441 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:04.745124 17441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:04.745332 17441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:04.747483 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "235f31fc84264e8c83fb9b575a730d22"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "481fb72b1da6f30f8a8a490274b33669"
server_key_iv: "701b383bcfcd0e389297251f655f5227"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.748044 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "235f31fc84264e8c83fb9b575a730d22"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "481fb72b1da6f30f8a8a490274b33669"
server_key_iv: "701b383bcfcd0e389297251f655f5227"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.751910 17441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 03:30:04.754371 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.755551 17441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:04.755661 17441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "235f31fc84264e8c83fb9b575a730d22"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "481fb72b1da6f30f8a8a490274b33669"
server_key_iv: "701b383bcfcd0e389297251f655f5227"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.755755 17441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:04.772693 17441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:04.775524 17441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:04.775704 17441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:04.776315 17441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:04.777163 17441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:04.777211 17441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.777266 17441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:04.777292 17441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.785897 17441 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:46265
I20260809 03:30:04.785920 17658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:46265 every 8 connection(s)
I20260809 03:30:04.786741 17441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:04.787791 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17441
I20260809 03:30:04.787894 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:04.788185 16249 external_mini_cluster.cc:1428] Setting key 62359d01378cd925a0a063285e991c43
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:04.800093 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.788527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:42063 (local address 127.15.222.126:32811)
0809 03:30:04.788678 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:04.788683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:04.789677 (+ 994us) server_negotiation.cc:407] Connection header received
0809 03:30:04.790755 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:04.790758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:04.790813 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:04.790897 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:04.791991 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.792505 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.793607 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.793927 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.794030 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.796639 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:04.796642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:04.796644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:04.796668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:04.797996 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.798587 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.798590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.798591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.798675 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:04.799054 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:04.799056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:04.799057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:04.799220 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:04.799291 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:04.799873 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 03:30:04.799971 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:04.800830 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.789007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:32811 (local address 127.15.222.66:42063)
0809 03:30:04.789504 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:04.789565 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:04.790555 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:04.791040 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:04.791047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:04.791373 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:04.791849 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.791863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.792622 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.792626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.793486 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.793496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.794071 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.794074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.794220 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.794770 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:04.794786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:04.796474 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:04.798108 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:04.798114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:04.798128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:04.798451 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:04.798770 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:04.798773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:04.798775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:04.798964 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:04.799311 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:04.799317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:04.799607 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 03:30:04.799815 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 03:30:04.800081 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260809 03:30:04.801932 17659 heartbeater.cc:347] Connected to a master server at 127.15.222.126:32811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
I20260809 03:30:04.802145 17659 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:04.802584 17659 heartbeater.cc:511] Master 127.15.222.126:32811 requested a full tablet report, sending...
I20260809 03:30:04.803587 17149 ts_manager.cc:194] Registered new tserver with Master: 235f31fc84264e8c83fb9b575a730d22 (127.15.222.66:46265)
I20260809 03:30:04.804327 17149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:42063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:04.816335 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:04.927337 17671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:04.927588 17671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:04.927637 17671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:04.931609 17671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:04.931663 17671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:04.931725 17671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:04.936828 17671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:32811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:04.937872 17671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:04.938862 17671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:04.948072 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.948170 17671 server_base.cc:1034] running on GCE node
W20260809 03:30:04.948081 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:04.948081 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:04.948776 17671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:04.949311 17671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:04.950454 17671 hybrid_clock.cc:648] HybridClock initialized: now 1786246204950435 us; error 31 us; skew 500 ppm
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:04.952960 17671 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:04.953775 17671 webserver.cc:460] Webserver started at http://127.15.222.67:37735/ using document root <none> and password file <none>
I20260809 03:30:04.954285 17671 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:04.954349 17671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:04.954559 17671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:04.956501 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "9a29e1e3969d9c2a710a2e4138c8112d"
server_key_iv: "7f42d6280d513fcd9d056be9ee1c6bd2"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.957027 17671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "9a29e1e3969d9c2a710a2e4138c8112d"
server_key_iv: "7f42d6280d513fcd9d056be9ee1c6bd2"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.960650 17671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:30:04.962875 17695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.964030 17671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:04.964138 17671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07"
format_stamp: "Formatted at 2026-08-09 03:30:04 on dist-test-slave-cvwc"
server_key: "9a29e1e3969d9c2a710a2e4138c8112d"
server_key_iv: "7f42d6280d513fcd9d056be9ee1c6bd2"
server_key_version: "encryptionkey@0"
I20260809 03:30:04.964223 17671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:04.978962 17671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:04.981606 17671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:04.981809 17671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:04.982407 17671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:04.983214 17671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:04.983260 17671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.983312 17671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:04.983338 17671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:04.991569 17671 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:33563
I20260809 03:30:04.991600 17880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:33563 every 8 connection(s)
I20260809 03:30:04.992412 17671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:30:04 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:05.002485 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17671
I20260809 03:30:05.002606 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:05.002877 16249 external_mini_cluster.cc:1428] Setting key b003cbc9bcb7b6005b20046b12e23b07
I20260809 03:30:05.004477 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.994100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:48567 (local address 127.15.222.126:32811)
0809 03:30:04.994219 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:04.994222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:04.995019 (+ 797us) server_negotiation.cc:407] Connection header received
0809 03:30:04.995818 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:04.995822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:04.995872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:04.995947 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:04.996999 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.997509 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.998600 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.998798 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.998867 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.001104 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:05.001107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:05.001109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:05.001134 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:05.002476 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.002965 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.002969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.002971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.003062 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.003414 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.003418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.003419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.003646 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:05.003737 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.004248 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.004358 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:05.005429 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:04.994417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:32811 (local address 127.15.222.67:48567)
0809 03:30:04.994837 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:04.994877 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:04.995624 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:04.996095 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:04.996101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:04.996435 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:04.996878 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.996901 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.997593 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.997595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.998509 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:04.998519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:04.998915 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:04.998917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:04.999052 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:04.999524 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:04.999540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:05.000932 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:05.002590 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:05.002597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:05.002610 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:05.002862 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:05.003144 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:05.003146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:05.003147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:05.003317 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:05.003746 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:05.003751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:05.004004 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.004189 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.004507 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260809 03:30:05.006409 17881 heartbeater.cc:347] Connected to a master server at 127.15.222.126:32811
I20260809 03:30:05.006604 17881 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:05.007019 17881 heartbeater.cc:511] Master 127.15.222.126:32811 requested a full tablet report, sending...
I20260809 03:30:05.007946 17149 ts_manager.cc:194] Registered new tserver with Master: 74821ce93ffd47a8b1f1b8c4d1e7cf07 (127.15.222.67:33563)
I20260809 03:30:05.008493 17149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:48567
I20260809 03:30:05.016626 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:05.026564 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.019984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.15.222.126:32811)
0809 03:30:05.020125 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.020130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.020260 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:30:05.020354 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.020356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.020394 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.020456 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:05.021094 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.021585 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.022347 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.022543 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.022606 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.023721 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:05.023724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:05.023726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:05.023748 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:05.025092 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.025461 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.025463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.025465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.025558 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.025805 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.025809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.025811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.026037 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:05.026149 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.026303 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.026393 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:30:05.030853 17149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:05.033665 17149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:05.047772 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.041531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37771 (local address 127.0.0.1:45854)
0809 03:30:05.042810 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:05.042823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:05.042903 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:05.043374 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:05.043376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:05.043404 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:05.043503 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.043508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.044688 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.044691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.046159 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.046170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.047230 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.047233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.047348 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.047404 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.047507 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.047624 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":38,"thread_start_us":78,"threads_started":1}
I20260809 03:30:05.047797 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.042283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.15.222.65:37771)
0809 03:30:05.042924 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.042937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.042962 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:30:05.043034 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.043039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.043246 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.043408 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:05.043614 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.044582 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.046296 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.047108 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.047217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.047357 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.047548 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.047644 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":149,"threads_started":1}
I20260809 03:30:05.047919 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.041857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:33563 (local address 127.0.0.1:60334)
0809 03:30:05.043599 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:05.043611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:05.043764 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:05.044242 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:05.044246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:05.044265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:05.044342 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.044347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.045706 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.045729 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.046667 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.046677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.047509 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.047513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.047598 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.047621 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.047692 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.047754 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":87,"threads_started":1}
I20260809 03:30:05.048389 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.042250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.15.222.67:33563)
0809 03:30:05.042691 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.042703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.043695 (+ 992us) server_negotiation.cc:407] Connection header received
0809 03:30:05.043908 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.043914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.044090 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.044233 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:05.044429 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.045551 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.046777 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.047381 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.047484 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.047618 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.047724 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.047812 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":114,"threads_started":1}
I20260809 03:30:05.048477 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.041464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:46265 (local address 127.0.0.1:57676)
0809 03:30:05.042244 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:05.042289 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:05.042419 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:05.043230 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:05.043236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:05.043258 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:05.043387 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.043394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.044720 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.044723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.046238 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.046246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.048108 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.048111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.048198 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.048220 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.048289 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.048352 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":94,"thread_start_us":82,"threads_started":1}
I20260809 03:30:05.048568 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.041610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.15.222.66:46265)
0809 03:30:05.042139 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.042152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.042559 (+ 407us) server_negotiation.cc:407] Connection header received
0809 03:30:05.042662 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.042667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.042877 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.043041 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:05.043509 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.044596 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.046358 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.047108 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.047217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.047357 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.048331 (+ 974us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.048437 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":149,"threads_started":1}
I20260809 03:30:05.050809 17339 tablet_service.cc:1467] Processing CreateTablet for tablet eca9e203e52e42a986c8062ef9af0c8e (DEFAULT_TABLE table=test-table [id=8dd61dbf289c40219919b0efc0872d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:05.051311 17779 tablet_service.cc:1467] Processing CreateTablet for tablet eca9e203e52e42a986c8062ef9af0c8e (DEFAULT_TABLE table=test-table [id=8dd61dbf289c40219919b0efc0872d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:05.052014 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca9e203e52e42a986c8062ef9af0c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:05.052313 17779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca9e203e52e42a986c8062ef9af0c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:05.059334 17922 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Bootstrap starting.
I20260809 03:30:05.059796 17549 tablet_service.cc:1467] Processing CreateTablet for tablet eca9e203e52e42a986c8062ef9af0c8e (DEFAULT_TABLE table=test-table [id=8dd61dbf289c40219919b0efc0872d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:05.060349 17923 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Bootstrap starting.
I20260809 03:30:05.060750 17549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca9e203e52e42a986c8062ef9af0c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:05.061831 17922 tablet_bootstrap.cc:654] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:05.062471 17923 tablet_bootstrap.cc:654] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:05.063151 17922 log.cc:824] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:05.063647 17923 log.cc:824] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:05.065475 17922 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: No bootstrap required, opened a new log
I20260809 03:30:05.065591 17923 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: No bootstrap required, opened a new log
I20260809 03:30:05.065671 17922 ts_tablet_manager.cc:1397] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 03:30:05.065762 17923 ts_tablet_manager.cc:1397] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:30:05.067337 17928 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Bootstrap starting.
I20260809 03:30:05.069594 17928 tablet_bootstrap.cc:654] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:05.069588 17923 raft_consensus.cc:348] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.069589 17922 raft_consensus.cc:348] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.069900 17922 raft_consensus.cc:374] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:05.069963 17922 raft_consensus.cc:729] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74821ce93ffd47a8b1f1b8c4d1e7cf07, State: Initialized, Role: FOLLOWER
I20260809 03:30:05.070003 17923 raft_consensus.cc:374] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:05.070075 17923 raft_consensus.cc:729] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed658b0201b844149b39455d713558ef, State: Initialized, Role: FOLLOWER
I20260809 03:30:05.070423 17923 consensus_queue.cc:260] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.070423 17922 consensus_queue.cc:260] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.070842 17928 log.cc:824] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:05.071420 17923 ts_tablet_manager.cc:1428] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:30:05.071409 17922 ts_tablet_manager.cc:1428] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260809 03:30:05.071787 17435 heartbeater.cc:503] Master 127.15.222.126:32811 was elected leader, sending a full tablet report...
I20260809 03:30:05.072747 17928 tablet_bootstrap.cc:492] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: No bootstrap required, opened a new log
I20260809 03:30:05.072808 17881 heartbeater.cc:503] Master 127.15.222.126:32811 was elected leader, sending a full tablet report...
I20260809 03:30:05.072939 17928 ts_tablet_manager.cc:1397] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260809 03:30:05.076514 17928 raft_consensus.cc:348] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.076754 17928 raft_consensus.cc:374] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:05.076812 17928 raft_consensus.cc:729] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235f31fc84264e8c83fb9b575a730d22, State: Initialized, Role: FOLLOWER
I20260809 03:30:05.077224 17928 consensus_queue.cc:260] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.077849 17659 heartbeater.cc:503] Master 127.15.222.126:32811 was elected leader, sending a full tablet report...
I20260809 03:30:05.078148 17928 ts_tablet_manager.cc:1428] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260809 03:30:05.243304 17887 tablet.cc:2369] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:05.256906 17929 raft_consensus.cc:482] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:05.257124 17929 raft_consensus.cc:504] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.258309 17929 leader_election.cc:288] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 235f31fc84264e8c83fb9b575a730d22 (127.15.222.66:46265), ed658b0201b844149b39455d713558ef (127.15.222.65:37771)
I20260809 03:30:05.262598 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.258650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37435 (local address 127.15.222.65:37771)
0809 03:30:05.258787 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.258791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.258806 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:05.258861 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.258864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.258928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.259005 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:05.259358 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.260080 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.261567 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.262124 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.262238 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.262291 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.262419 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.262488 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:30:05.262616 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.258576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37771 (local address 127.15.222.67:37435)
0809 03:30:05.258675 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:05.258689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:05.258783 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:05.259063 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:05.259066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:05.259108 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:05.259233 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.259240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.260178 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.260181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.261449 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.261459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.262223 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.262226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.262306 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.262327 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.262408 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.262481 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":20}
I20260809 03:30:05.262784 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.258575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:46265 (local address 127.15.222.67:35767)
0809 03:30:05.259190 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:05.259205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:05.259301 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:05.259543 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:05.259545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:05.259558 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:05.259657 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.259661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.260496 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.260499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.261651 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:05.261658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.262421 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.262424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:05.262519 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.262541 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:05.262610 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:05.262670 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":319,"thread_start_us":126,"threads_started":1}
I20260809 03:30:05.262820 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.258670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:35767 (local address 127.15.222.66:46265)
0809 03:30:05.258787 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.258790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.259283 (+ 493us) server_negotiation.cc:407] Connection header received
0809 03:30:05.259398 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.259401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.259454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.259525 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:05.259740 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.260372 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.261762 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.262315 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.262421 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.262464 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.262632 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.262701 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:05.263306 17371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca9e203e52e42a986c8062ef9af0c8e" candidate_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed658b0201b844149b39455d713558ef" is_pre_election: true
I20260809 03:30:05.263408 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca9e203e52e42a986c8062ef9af0c8e" candidate_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235f31fc84264e8c83fb9b575a730d22" is_pre_election: true
I20260809 03:30:05.263608 17371 raft_consensus.cc:2454] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74821ce93ffd47a8b1f1b8c4d1e7cf07 in term 0.
I20260809 03:30:05.263703 17583 raft_consensus.cc:2454] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74821ce93ffd47a8b1f1b8c4d1e7cf07 in term 0.
I20260809 03:30:05.264124 17702 leader_election.cc:302] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 235f31fc84264e8c83fb9b575a730d22, 74821ce93ffd47a8b1f1b8c4d1e7cf07, ed658b0201b844149b39455d713558ef; no voters:
I20260809 03:30:05.264365 17929 raft_consensus.cc:2781] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:05.264451 17929 raft_consensus.cc:482] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:05.264498 17929 raft_consensus.cc:3037] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:05.265487 17929 raft_consensus.cc:504] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.265815 17929 leader_election.cc:288] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [CANDIDATE]: Term 1 election: Requested vote from peers 235f31fc84264e8c83fb9b575a730d22 (127.15.222.66:46265), ed658b0201b844149b39455d713558ef (127.15.222.65:37771)
I20260809 03:30:05.266121 17371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca9e203e52e42a986c8062ef9af0c8e" candidate_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed658b0201b844149b39455d713558ef"
I20260809 03:30:05.266126 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca9e203e52e42a986c8062ef9af0c8e" candidate_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235f31fc84264e8c83fb9b575a730d22"
I20260809 03:30:05.266223 17583 raft_consensus.cc:3037] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:05.266234 17371 raft_consensus.cc:3037] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:05.267274 17371 raft_consensus.cc:2454] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74821ce93ffd47a8b1f1b8c4d1e7cf07 in term 1.
I20260809 03:30:05.267367 17583 raft_consensus.cc:2454] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74821ce93ffd47a8b1f1b8c4d1e7cf07 in term 1.
I20260809 03:30:05.267563 17702 leader_election.cc:302] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74821ce93ffd47a8b1f1b8c4d1e7cf07, ed658b0201b844149b39455d713558ef; no voters:
I20260809 03:30:05.267773 17929 raft_consensus.cc:2781] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:05.267977 17929 raft_consensus.cc:686] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [term 1 LEADER]: Becoming Leader. State: Replica: 74821ce93ffd47a8b1f1b8c4d1e7cf07, State: Running, Role: LEADER
I20260809 03:30:05.268301 17929 consensus_queue.cc:237] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } }
I20260809 03:30:05.271452 17146 catalog_manager.cc:5495] T eca9e203e52e42a986c8062ef9af0c8e P 74821ce93ffd47a8b1f1b8c4d1e7cf07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74821ce93ffd47a8b1f1b8c4d1e7cf07 (127.15.222.67). New cstate: current_term: 1 leader_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235f31fc84264e8c83fb9b575a730d22" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed658b0201b844149b39455d713558ef" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74821ce93ffd47a8b1f1b8c4d1e7cf07" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33563 } health_report { overall_health: HEALTHY } } }
W20260809 03:30:05.287949 17662 tablet.cc:2369] T eca9e203e52e42a986c8062ef9af0c8e P 235f31fc84264e8c83fb9b575a730d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:30:05.320196 17436 tablet.cc:2369] T eca9e203e52e42a986c8062ef9af0c8e P ed658b0201b844149b39455d713558ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:05.338039 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.330024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.15.222.65:37771)
0809 03:30:05.330214 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.330219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.330337 (+ 118us) server_negotiation.cc:407] Connection header received
0809 03:30:05.330452 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.330454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.330497 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.330564 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:05.331197 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.331681 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.332433 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.332638 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.332700 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.335028 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:05.335040 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:05.335056 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:05.335080 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:05.336558 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.336953 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.336958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.336959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.337043 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.337226 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.337228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.337230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.337465 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:05.337609 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.337811 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.337920 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:05.363808 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.356059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.15.222.65:37771)
0809 03:30:05.356174 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.356179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.356324 (+ 145us) server_negotiation.cc:407] Connection header received
0809 03:30:05.356480 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.356483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.356525 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.356588 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:05.357228 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.357788 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.358634 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.358858 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.358946 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.361130 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:05.361133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:05.361135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:05.361157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:05.362301 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.362667 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.362671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.362672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.362747 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.362984 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.362988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.362990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.363145 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:05.363272 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.363545 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.363658 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:05.387709 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:05.380336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.15.222.65:37771)
0809 03:30:05.380450 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:05.380455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:05.380601 (+ 146us) server_negotiation.cc:407] Connection header received
0809 03:30:05.380727 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:05.380731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:05.380773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:05.380846 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:05.381464 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.382018 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.382808 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:05.383019 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:05.383104 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:05.385185 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:05.385188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:05.385190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:05.385212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:05.386278 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.386641 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.386644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.386645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.386717 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:05.386956 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:05.386960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:05.386962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:05.387117 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:05.387227 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:05.387452 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 03:30:05.387562 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 03:30:05.388989 17313 tablet_service.cc:2908] Rejecting scan request for tablet eca9e203e52e42a986c8062ef9af0c8e: Uninitialized: clean time has not yet been initialized
I20260809 03:30:05.390911 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17244
I20260809 03:30:05.407757 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17441
I20260809 03:30:05.431520 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17671
I20260809 03:30:05.444548 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17104
2026-08-09T03:30:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2591 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17958](info): set up 2 sockets
Aug 09 03:30:05 dist-test-slave-cvwc krb5kdc[17958](info): commencing operation
krb5kdc: starting...
W20260809 03:30:06.584299 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:06 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:06.701189 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:36021
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:36021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:06.809376 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:06.809615 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:06.809667 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:06.813539 17982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:06.813592 17982 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:30:06.813614 17982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:06.813637 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:06.813666 17982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:06.813694 17982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:06.818838 17982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:36021
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:36021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:06.819808 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:06.820819 17982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:06.828713 17991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:06.828734 17990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:06.828734 17997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:06.829069 17982 server_base.cc:1034] running on GCE node
I20260809 03:30:06.829620 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:06.830667 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:06.831821 17982 hybrid_clock.cc:648] HybridClock initialized: now 1786246206831803 us; error 30 us; skew 500 ppm
Aug 09 03:30:06 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246206, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:06.834504 17982 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:06.835337 17982 webserver.cc:460] Webserver started at http://127.15.222.126:41989/ using document root <none> and password file <none>
I20260809 03:30:06.835858 17982 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:06.835937 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:06.836138 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:06.838160 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3e570e22f9fd43cbadd846b71bf981b1"
format_stamp: "Formatted at 2026-08-09 03:30:06 on dist-test-slave-cvwc"
server_key: "8a08989e5525b61c1db8fcde12470e70"
server_key_iv: "1598f21691d90ca7c5399e24ef3fdab1"
server_key_version: "encryptionkey@0"
I20260809 03:30:06.838692 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3e570e22f9fd43cbadd846b71bf981b1"
format_stamp: "Formatted at 2026-08-09 03:30:06 on dist-test-slave-cvwc"
server_key: "8a08989e5525b61c1db8fcde12470e70"
server_key_iv: "1598f21691d90ca7c5399e24ef3fdab1"
server_key_version: "encryptionkey@0"
I20260809 03:30:06.842190 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:30:06.844553 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:06.846060 17982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:06.846191 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3e570e22f9fd43cbadd846b71bf981b1"
format_stamp: "Formatted at 2026-08-09 03:30:06 on dist-test-slave-cvwc"
server_key: "8a08989e5525b61c1db8fcde12470e70"
server_key_iv: "1598f21691d90ca7c5399e24ef3fdab1"
server_key_version: "encryptionkey@0"
I20260809 03:30:06.846328 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:06.862345 17982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:06.864923 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:06.865072 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:06.872480 17982 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:36021
I20260809 03:30:06.872489 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:36021 every 8 connection(s)
I20260809 03:30:06.873382 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:06.876081 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:06.876971 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17982
I20260809 03:30:06.877070 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:06.877317 16249 external_mini_cluster.cc:1428] Setting key a022b2b47f0f9c363792d6f4386d245a
I20260809 03:30:06.882000 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Bootstrap starting.
I20260809 03:30:06.884136 18116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:06.884825 18116 log.cc:824] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:30:06 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:06.886765 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: No bootstrap required, opened a new log
I20260809 03:30:06.889932 18116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } }
I20260809 03:30:06.890200 18116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:06.890275 18116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e570e22f9fd43cbadd846b71bf981b1, State: Initialized, Role: FOLLOWER
I20260809 03:30:06.890705 18116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } }
I20260809 03:30:06.890827 18116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:06.890885 18116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:06.890977 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:06.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.15.222.126:36021)
0809 03:30:06.879124 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:06.879139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:06.879173 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:06.879804 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:06.879821 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:06.880132 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:06.880435 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:06.881195 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:06.882238 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:06.883058 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:06.883509 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:06.883655 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:06.886060 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:06.886065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:06.886078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:06.886104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:06.888313 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:06.888796 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:06.888801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:06.888805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:06.888908 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:06.889153 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:06.889155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:06.889157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:06.889504 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:06.889647 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:06.889944 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 03:30:06.890202 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":147,"threads_started":1}
I20260809 03:30:06.890981 18116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:06.891844 18116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } }
I20260809 03:30:06.892158 18116 leader_election.cc:302] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e570e22f9fd43cbadd846b71bf981b1; no voters:
I20260809 03:30:06.892433 18116 leader_election.cc:288] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:06.892503 18127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:06.892678 18127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e570e22f9fd43cbadd846b71bf981b1, State: Running, Role: LEADER
I20260809 03:30:06.892975 18127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } }
I20260809 03:30:06.893669 18116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:06.894119 18131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e570e22f9fd43cbadd846b71bf981b1. Latest consensus state: current_term: 1 leader_uuid: "3e570e22f9fd43cbadd846b71bf981b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } } }
I20260809 03:30:06.894138 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e570e22f9fd43cbadd846b71bf981b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e570e22f9fd43cbadd846b71bf981b1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 36021 } } }
I20260809 03:30:06.894248 18131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:06.894299 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:06.894622 18135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:06.897650 18135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:06.902931 18135 catalog_manager.cc:1347] Generated new cluster ID: 9578a61bfda14b5483e4370f9cd29a03
I20260809 03:30:06.902992 18135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:06.932516 18135 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:06.933466 18135 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:06.937280 18135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Generated new TSK 0
I20260809 03:30:06.937988 18135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:06.966216 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:36021
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:07.075213 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:07.075476 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:07.075537 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:07.079480 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:07.079536 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:07.079589 18159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:07.079644 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:07.084709 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:36021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:07.085752 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:07.086758 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:07.095918 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:07.096086 18159 server_base.cc:1034] running on GCE node
W20260809 03:30:07.095927 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:07.095973 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:07.096625 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:07.097198 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:07.098378 18159 hybrid_clock.cc:648] HybridClock initialized: now 1786246207098357 us; error 30 us; skew 500 ppm
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:07.101030 18159 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:07.101867 18159 webserver.cc:460] Webserver started at http://127.15.222.65:46553/ using document root <none> and password file <none>
I20260809 03:30:07.102362 18159 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:07.102417 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:07.102576 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:07.104466 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "24f0ad004bfb4540bcb67809e165cefa"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "37ff61514317c91e2763850e5db8212a"
server_key_iv: "177fcfadfdaf15c76737993b7324406e"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.104952 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "24f0ad004bfb4540bcb67809e165cefa"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "37ff61514317c91e2763850e5db8212a"
server_key_iv: "177fcfadfdaf15c76737993b7324406e"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.108352 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:30:07.110634 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.111866 18159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:07.111974 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "24f0ad004bfb4540bcb67809e165cefa"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "37ff61514317c91e2763850e5db8212a"
server_key_iv: "177fcfadfdaf15c76737993b7324406e"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.112062 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:07.135273 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:07.138366 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:07.138553 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:07.139900 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:07.139948 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.140002 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:07.140026 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:07.149415 18159 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:35291
I20260809 03:30:07.149734 18351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:35291 every 8 connection(s)
I20260809 03:30:07.150404 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:07.152453 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18159
I20260809 03:30:07.152545 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.140616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:34093 (local address 127.15.222.126:36021)
0809 03:30:07.140785 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.140791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.141578 (+ 787us) server_negotiation.cc:407] Connection header received
0809 03:30:07.142649 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.142654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.142712 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.142767 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.144351 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.144965 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.146345 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.146585 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.146674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.149097 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.149101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.149104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.149132 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.150602 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.151218 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.151221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.151223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.151314 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.151610 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.151612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.151614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.151773 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.151856 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.152308 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.152418 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 03:30:07.152570 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:07.152845 16249 external_mini_cluster.cc:1428] Setting key 1dd54b7b693de3340d49af2477920b00
I20260809 03:30:07.153436 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.140988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.65:34093)
0809 03:30:07.141426 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.141475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.142406 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.143184 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.143192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.143558 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.144206 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.144227 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.145089 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.145093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.146226 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.146237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.146727 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.146731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.146912 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.147402 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.147418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.148946 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.150710 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.150715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.150752 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.151083 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.151394 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.151397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.151398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.151517 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.151866 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.151872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.152083 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.152238 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.152547 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260809 03:30:07.156050 18353 heartbeater.cc:347] Connected to a master server at 127.15.222.126:36021
I20260809 03:30:07.156183 18353 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:07.156605 18353 heartbeater.cc:511] Master 127.15.222.126:36021 requested a full tablet report, sending...
I20260809 03:30:07.158139 18048 ts_manager.cc:194] Registered new tserver with Master: 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291)
I20260809 03:30:07.159348 18048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:34093
I20260809 03:30:07.163551 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.156597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.65:38185)
0809 03:30:07.156918 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.156932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.157025 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.157310 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.157312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.157513 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.157600 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.157606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.158561 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.158564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.159096 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.159104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.159605 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.159608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.159675 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.160099 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.160110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.160500 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.162311 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.162314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.162315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.162506 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.162760 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.162762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.162763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.162824 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.163178 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.163180 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.163244 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.163312 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.163414 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260809 03:30:07.163640 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.156660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:38185 (local address 127.15.222.126:36021)
0809 03:30:07.156757 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.156761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.157010 (+ 249us) server_negotiation.cc:407] Connection header received
0809 03:30:07.157098 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.157101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.157145 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.157212 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.157705 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.158437 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.159201 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.159503 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.159602 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.160634 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.160638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.160641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.160664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.162216 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.162594 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.162597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.162599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.162683 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.162898 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.162901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.162902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.163099 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.163200 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.163383 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.163500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:07.181821 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:36021
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:07.289410 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:07.289646 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:07.289688 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:07.293612 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:07.293659 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:07.293699 18379 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:07.293768 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:07.299217 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:36021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:07.300177 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:07.301143 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:07.310026 18387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:07.310032 18384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:07.310098 18385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:07.310200 18379 server_base.cc:1034] running on GCE node
I20260809 03:30:07.310657 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:07.311210 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:07.312359 18379 hybrid_clock.cc:648] HybridClock initialized: now 1786246207312337 us; error 32 us; skew 500 ppm
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:07.314988 18379 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:07.315800 18379 webserver.cc:460] Webserver started at http://127.15.222.66:38701/ using document root <none> and password file <none>
I20260809 03:30:07.316296 18379 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:07.316350 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:07.316531 18379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:07.318555 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "5795897d1ca74405ac3083d9ad1af4d7"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "054326a60da819bfad8ebdb2bfb17e53"
server_key_iv: "5a32b7a653718806194f26fb6c1c3273"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.319087 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "5795897d1ca74405ac3083d9ad1af4d7"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "054326a60da819bfad8ebdb2bfb17e53"
server_key_iv: "5a32b7a653718806194f26fb6c1c3273"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.322537 18379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:07.324688 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.325816 18379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:07.325920 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "5795897d1ca74405ac3083d9ad1af4d7"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "054326a60da819bfad8ebdb2bfb17e53"
server_key_iv: "5a32b7a653718806194f26fb6c1c3273"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.326006 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:07.335748 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:07.338224 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:07.338375 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:07.339677 18379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:07.339722 18379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.339761 18379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:07.339789 18379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:07.349334 18379 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42587
I20260809 03:30:07.349402 18596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42587 every 8 connection(s)
I20260809 03:30:07.350337 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:07.352898 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.340408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:42645 (local address 127.15.222.126:36021)
0809 03:30:07.340559 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.340564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.341321 (+ 757us) server_negotiation.cc:407] Connection header received
0809 03:30:07.342320 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.342324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.342384 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.342445 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.344037 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.344851 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.346475 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.346797 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.346950 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.349741 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.349745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.349749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.349776 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.351240 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.351721 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.351724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.351725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.351803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.352071 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.352073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.352074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.352230 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.352300 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.352678 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.352769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:07.353605 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.340725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.66:42645)
0809 03:30:07.341161 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.341209 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.342076 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.342845 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.342854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.343220 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.343824 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.343842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.344985 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.344989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.346348 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.346360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.346999 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.347003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.347140 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.347692 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.347714 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.349561 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.351349 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.351354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.351368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.351632 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.351870 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.351872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.351873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.352007 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.352298 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.352302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.352505 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.352645 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.352865 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260809 03:30:07.354727 18597 heartbeater.cc:347] Connected to a master server at 127.15.222.126:36021
I20260809 03:30:07.354856 18597 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:07.355296 18597 heartbeater.cc:511] Master 127.15.222.126:36021 requested a full tablet report, sending...
I20260809 03:30:07.356366 18049 ts_manager.cc:194] Registered new tserver with Master: 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587)
I20260809 03:30:07.357069 18049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:42645
I20260809 03:30:07.357247 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18379
I20260809 03:30:07.357354 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:07.357630 16249 external_mini_cluster.cc:1428] Setting key 2f690c8c2782339587a49798959b5479
I20260809 03:30:07.363173 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.356604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34203 (local address 127.15.222.126:36021)
0809 03:30:07.356710 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.356714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.356892 (+ 178us) server_negotiation.cc:407] Connection header received
0809 03:30:07.356983 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.356985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.357025 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.357103 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.357687 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.358370 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.359348 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.359621 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.359718 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.360574 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.360577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.360580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.360604 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.361775 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.362184 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.362186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.362188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.362250 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.362483 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.362485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.362486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.362662 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.362733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.362987 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.363069 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:07.363212 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.356546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.66:34203)
0809 03:30:07.356811 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.356824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.356914 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.357128 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.357131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.357477 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.357589 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.357597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.358509 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.358513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.359233 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.359242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.359714 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.359717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.359776 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.360163 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.360177 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.360453 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.361874 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.361878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.361879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.362112 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.362317 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.362320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.362322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.362410 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.362746 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.362748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.362836 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.362938 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.363058 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:07.383934 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:36021
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:07.494311 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:07.494563 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:07.494608 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:07.498512 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:07.498562 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:07.498597 18611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:07.498632 18611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:07.503569 18611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:36021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:07.504444 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:07.505358 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:07.514369 18617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:07.514396 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:07.514415 18611 server_base.cc:1034] running on GCE node
W20260809 03:30:07.514361 18621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:07.514957 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:07.515581 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:07.516748 18611 hybrid_clock.cc:648] HybridClock initialized: now 1786246207516718 us; error 36 us; skew 500 ppm
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:07.519632 18611 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:07.520486 18611 webserver.cc:460] Webserver started at http://127.15.222.67:42033/ using document root <none> and password file <none>
I20260809 03:30:07.520985 18611 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:07.521046 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:07.521217 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:07.523253 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "3cee25a51c97455fb2a7047cddb10676"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "17fb4aac1d25438af92c7473017a4c1a"
server_key_iv: "66ef908c0d925979a2d9f23f07adddb1"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.523787 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "3cee25a51c97455fb2a7047cddb10676"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "17fb4aac1d25438af92c7473017a4c1a"
server_key_iv: "66ef908c0d925979a2d9f23f07adddb1"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.527397 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:07.529804 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.530929 18611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:07.531037 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "3cee25a51c97455fb2a7047cddb10676"
format_stamp: "Formatted at 2026-08-09 03:30:07 on dist-test-slave-cvwc"
server_key: "17fb4aac1d25438af92c7473017a4c1a"
server_key_iv: "66ef908c0d925979a2d9f23f07adddb1"
server_key_version: "encryptionkey@0"
I20260809 03:30:07.531137 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:07.546478 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:07.549707 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:07.549911 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:07.551406 18611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:07.551460 18611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:07.551505 18611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:07.551537 18611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:07.562945 18611 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:37417
I20260809 03:30:07.563843 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:07.564575 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:37417 every 8 connection(s)
I20260809 03:30:07.565493 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.552813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.67:33639)
0809 03:30:07.553265 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.553313 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.554315 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.555090 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.555099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.555574 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.556256 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.556278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.557127 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.557131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.558076 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.558086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.558575 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.558580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.558756 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.559322 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.559341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.561132 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.562912 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.562917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.562931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.563235 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.563494 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.563497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.563499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.563630 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.564001 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.564008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.564239 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.564436 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.564745 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260809 03:30:07.566139 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.551999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:33639 (local address 127.15.222.126:36021)
0809 03:30:07.552150 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.552156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.553423 (+ 1267us) server_negotiation.cc:407] Connection header received
0809 03:30:07.554511 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.554516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.554578 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.554704 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.556418 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.556998 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.558190 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.558412 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.558514 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.561314 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.561319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.561322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.561348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.562804 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.563325 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.563329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.563331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.563413 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.563715 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.563718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.563719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.563911 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.564001 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.565881 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.566003 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:30:07.567009 18815 heartbeater.cc:347] Connected to a master server at 127.15.222.126:36021
I20260809 03:30:07.567137 18815 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:07.567553 18815 heartbeater.cc:511] Master 127.15.222.126:36021 requested a full tablet report, sending...
I20260809 03:30:07.568485 18048 ts_manager.cc:194] Registered new tserver with Master: 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417)
I20260809 03:30:07.569067 18048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:33639
I20260809 03:30:07.569990 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18611
I20260809 03:30:07.570099 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:07.570379 16249 external_mini_cluster.cc:1428] Setting key 3dd16086370f69a0d3065e592b506630
I20260809 03:30:07.572517 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:07.575556 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.569075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:49235 (local address 127.15.222.126:36021)
0809 03:30:07.569363 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.569367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.569379 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:07.569421 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.569424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.569457 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.569522 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.569976 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.570565 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.571511 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.571795 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.571905 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.572829 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.572832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.572834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.572854 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.574108 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.574506 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.574509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.574510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.574572 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.574827 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.574830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.574832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.574998 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.575081 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.575320 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.575418 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260809 03:30:07.575568 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.568972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.15.222.67:49235)
0809 03:30:07.569242 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.569256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.569343 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.569570 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.569573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.569767 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:07.569869 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.569875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.570680 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.570684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.571410 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.571421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.571902 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.571904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.571974 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.572356 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:07.572367 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:07.572714 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:07.574202 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.574205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.574207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.574429 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.574660 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:07.574663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:07.574665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:07.574748 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:07.575087 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:07.575090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:07.575175 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.575257 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.575387 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:30:07 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:07.598031 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.590486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.15.222.126:36021)
0809 03:30:07.590615 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.590619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.590631 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:07.590673 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.590675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.590714 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.590784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:07.591558 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.592060 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.592857 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.593064 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.593132 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.595555 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:07.595558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:07.595561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:07.595586 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:07.596714 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.597083 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.597085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.597086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.597151 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:07.597329 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:07.597331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:07.597332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:07.597501 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:07.597595 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.597805 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.597908 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:07.600934 18048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:07.603996 18048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:07.618373 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.15.222.66:42587)
0809 03:30:07.612577 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.612587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.613709 (+ 1122us) server_negotiation.cc:407] Connection header received
0809 03:30:07.613971 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.613977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.614143 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.614289 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:07.614698 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615505 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.616939 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.617691 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.617827 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.618008 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.618128 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.618219 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
I20260809 03:30:07.618364 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42587 (local address 127.0.0.1:56390)
0809 03:30:07.613575 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.613611 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.613786 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.614443 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.614445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.614461 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:07.614565 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.614572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615657 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.615660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.616814 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.616824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.617831 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.617835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.617945 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.617983 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.618096 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.618213 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"spinlock_wait_cycles":3200,"thread_start_us":64,"threads_started":1}
I20260809 03:30:07.619110 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35291 (local address 127.0.0.1:40852)
0809 03:30:07.613574 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.613612 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.613785 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.614301 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.614307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.614329 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:07.614464 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.614472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615936 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.615940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.617615 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.617622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.618738 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.618741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.618828 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.618865 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.618910 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.618974 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":842,"spinlock_wait_cycles":17408,"thread_start_us":88,"threads_started":1}
I20260809 03:30:07.619445 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37417 (local address 127.0.0.1:39216)
0809 03:30:07.613987 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.613998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.614086 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.614508 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.614511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.614525 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:07.614593 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.614599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615963 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.615965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.616872 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.616884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.618837 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.618840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.619173 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.619196 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.619263 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.619323 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":277,"thread_start_us":72,"threads_started":1}
I20260809 03:30:07.619742 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.15.222.65:35291)
0809 03:30:07.613339 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.613354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.613699 (+ 345us) server_negotiation.cc:407] Connection header received
0809 03:30:07.613936 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.613942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.614143 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.614289 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:07.614591 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615801 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.617746 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.618596 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.619278 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.619457 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.619510 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.619604 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":96,"threads_started":1}
I20260809 03:30:07.619935 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.612998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.15.222.67:37417)
0809 03:30:07.613366 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.613380 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.614087 (+ 707us) server_negotiation.cc:407] Connection header received
0809 03:30:07.614180 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.614185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.614377 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.614533 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:07.614663 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.615857 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.617773 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.618725 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.618880 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.619051 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.619697 (+ 646us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.619793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":99,"threads_started":1}
I20260809 03:30:07.621389 18487 tablet_service.cc:1467] Processing CreateTablet for tablet b8464015f8fd4f3890c596a3254bf18c (DEFAULT_TABLE table=test-table [id=4a46b31238474de2a7c36749caecdfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.622080 18256 tablet_service.cc:1467] Processing CreateTablet for tablet b8464015f8fd4f3890c596a3254bf18c (DEFAULT_TABLE table=test-table [id=4a46b31238474de2a7c36749caecdfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.622464 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8464015f8fd4f3890c596a3254bf18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.622934 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8464015f8fd4f3890c596a3254bf18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.628473 18715 tablet_service.cc:1467] Processing CreateTablet for tablet b8464015f8fd4f3890c596a3254bf18c (DEFAULT_TABLE table=test-table [id=4a46b31238474de2a7c36749caecdfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.629148 18860 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Bootstrap starting.
I20260809 03:30:07.629462 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8464015f8fd4f3890c596a3254bf18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.629973 18861 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Bootstrap starting.
I20260809 03:30:07.631814 18860 tablet_bootstrap.cc:654] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.632031 18861 tablet_bootstrap.cc:654] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.632529 18860 log.cc:824] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:07.632823 18861 log.cc:824] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:07.634399 18860 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: No bootstrap required, opened a new log
I20260809 03:30:07.634663 18860 ts_tablet_manager.cc:1397] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260809 03:30:07.634786 18861 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: No bootstrap required, opened a new log
I20260809 03:30:07.634970 18861 ts_tablet_manager.cc:1397] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:07.636281 18864 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Bootstrap starting.
I20260809 03:30:07.638105 18864 tablet_bootstrap.cc:654] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.638528 18860 raft_consensus.cc:348] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.638726 18864 log.cc:824] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:07.638788 18860 raft_consensus.cc:374] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.638852 18860 raft_consensus.cc:729] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5795897d1ca74405ac3083d9ad1af4d7, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.638931 18861 raft_consensus.cc:348] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.639159 18861 raft_consensus.cc:374] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.639212 18861 raft_consensus.cc:729] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f0ad004bfb4540bcb67809e165cefa, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.639287 18860 consensus_queue.cc:260] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.639600 18861 consensus_queue.cc:260] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.639914 18597 heartbeater.cc:503] Master 127.15.222.126:36021 was elected leader, sending a full tablet report...
I20260809 03:30:07.640198 18353 heartbeater.cc:503] Master 127.15.222.126:36021 was elected leader, sending a full tablet report...
I20260809 03:30:07.640260 18860 ts_tablet_manager.cc:1428] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260809 03:30:07.640434 18861 ts_tablet_manager.cc:1428] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:30:07.642009 18864 tablet_bootstrap.cc:492] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: No bootstrap required, opened a new log
I20260809 03:30:07.642174 18864 ts_tablet_manager.cc:1397] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:30:07.645373 18864 raft_consensus.cc:348] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.645555 18864 raft_consensus.cc:374] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.645591 18864 raft_consensus.cc:729] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cee25a51c97455fb2a7047cddb10676, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.646132 18864 consensus_queue.cc:260] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.646682 18815 heartbeater.cc:503] Master 127.15.222.126:36021 was elected leader, sending a full tablet report...
I20260809 03:30:07.646858 18864 ts_tablet_manager.cc:1428] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260809 03:30:07.651584 18356 tablet.cc:2369] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:07.742937 18866 raft_consensus.cc:482] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:07.743149 18866 raft_consensus.cc:504] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.744238 18866 leader_election.cc:288] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291), 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417)
I20260809 03:30:07.748174 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.744629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:36965 (local address 127.15.222.67:37417)
0809 03:30:07.744748 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.744752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.745001 (+ 249us) server_negotiation.cc:407] Connection header received
0809 03:30:07.745139 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.745142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.745192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.745262 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:07.745504 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.746063 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.747116 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.747679 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.747766 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.747807 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.748024 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.748075 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:07.748234 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.744474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37417 (local address 127.15.222.66:36965)
0809 03:30:07.744908 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.744921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.745041 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.745274 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.745277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.745293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:07.745409 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.745415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.746173 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.746176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.746988 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.746997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.747784 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.747788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.747906 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.747934 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.748010 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.748076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":104,"spinlock_wait_cycles":6400,"thread_start_us":100,"threads_started":1}
I20260809 03:30:07.748642 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.744456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35291 (local address 127.15.222.66:56615)
0809 03:30:07.744590 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:07.744605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:07.744692 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:07.744973 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:07.744976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:07.745033 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:07.745153 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.745160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.745988 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.745992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.747476 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:07.747487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.748255 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.748259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:07.748367 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.748388 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:30:07.748468 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 03:30:07.748530 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":42624}
I20260809 03:30:07.748669 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.744563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:56615 (local address 127.15.222.65:35291)
0809 03:30:07.744672 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.744676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.744690 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:07.744820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.744823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.744872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.744948 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:07.745277 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.745886 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.747609 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.748138 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.748245 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.748291 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.748491 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.748565 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:07.748811 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8464015f8fd4f3890c596a3254bf18c" candidate_uuid: "5795897d1ca74405ac3083d9ad1af4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee25a51c97455fb2a7047cddb10676" is_pre_election: true
I20260809 03:30:07.749126 18750 raft_consensus.cc:2454] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5795897d1ca74405ac3083d9ad1af4d7 in term 0.
I20260809 03:30:07.749241 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8464015f8fd4f3890c596a3254bf18c" candidate_uuid: "5795897d1ca74405ac3083d9ad1af4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f0ad004bfb4540bcb67809e165cefa" is_pre_election: true
I20260809 03:30:07.749516 18290 raft_consensus.cc:2454] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5795897d1ca74405ac3083d9ad1af4d7 in term 0.
I20260809 03:30:07.749616 18404 leader_election.cc:302] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cee25a51c97455fb2a7047cddb10676, 5795897d1ca74405ac3083d9ad1af4d7; no voters:
I20260809 03:30:07.749903 18866 raft_consensus.cc:2781] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:07.749980 18866 raft_consensus.cc:482] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:07.750026 18866 raft_consensus.cc:3037] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:07.751104 18866 raft_consensus.cc:504] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.751420 18866 leader_election.cc:288] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [CANDIDATE]: Term 1 election: Requested vote from peers 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291), 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417)
I20260809 03:30:07.751751 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8464015f8fd4f3890c596a3254bf18c" candidate_uuid: "5795897d1ca74405ac3083d9ad1af4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f0ad004bfb4540bcb67809e165cefa"
I20260809 03:30:07.751855 18290 raft_consensus.cc:3037] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:07.752789 18290 raft_consensus.cc:2454] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5795897d1ca74405ac3083d9ad1af4d7 in term 1.
I20260809 03:30:07.753039 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8464015f8fd4f3890c596a3254bf18c" candidate_uuid: "5795897d1ca74405ac3083d9ad1af4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee25a51c97455fb2a7047cddb10676"
I20260809 03:30:07.753094 18403 leader_election.cc:302] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f0ad004bfb4540bcb67809e165cefa, 5795897d1ca74405ac3083d9ad1af4d7; no voters:
I20260809 03:30:07.753185 18750 raft_consensus.cc:3037] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:07.753242 18866 raft_consensus.cc:2781] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:07.753448 18866 raft_consensus.cc:686] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 LEADER]: Becoming Leader. State: Replica: 5795897d1ca74405ac3083d9ad1af4d7, State: Running, Role: LEADER
I20260809 03:30:07.753798 18866 consensus_queue.cc:237] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.754726 18750 raft_consensus.cc:2454] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5795897d1ca74405ac3083d9ad1af4d7 in term 1.
I20260809 03:30:07.757035 18047 catalog_manager.cc:5495] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "5795897d1ca74405ac3083d9ad1af4d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:07.784200 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:07.780634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.15.222.66:42587)
0809 03:30:07.780766 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:07.780771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:07.780786 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:07.780946 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:07.780949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:07.781004 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:07.781088 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:07.781341 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.781883 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.782783 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:07.783000 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:07.783086 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:07.783409 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:07.783802 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:07.783914 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:07.783991 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 03:30:07.784048 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:07.789701 18290 raft_consensus.cc:1261] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Refusing update from remote peer 5795897d1ca74405ac3083d9ad1af4d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:07.789701 18750 raft_consensus.cc:1261] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Refusing update from remote peer 5795897d1ca74405ac3083d9ad1af4d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:07.790388 18866 consensus_queue.cc:1035] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:07.790565 18876 consensus_queue.cc:1035] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:07.801277 18887 mvcc.cc:205] Tried to move back new op lower bound from 7316464467099717632 to 7316464466962550784. Current Snapshot: MvccSnapshot[applied={T|T < 7316464467099717632 or (T in {7316464467099717632})}]
I20260809 03:30:07.808769 18047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:07.808920 18047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:07.811132 18047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Sending DeleteTablet for 3 replicas of tablet b8464015f8fd4f3890c596a3254bf18c
I20260809 03:30:07.811836 18256 tablet_service.cc:1514] Processing DeleteTablet for tablet b8464015f8fd4f3890c596a3254bf18c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:40852
I20260809 03:30:07.811904 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet b8464015f8fd4f3890c596a3254bf18c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39216
I20260809 03:30:07.811949 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet b8464015f8fd4f3890c596a3254bf18c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:56390
I20260809 03:30:07.812361 18899 tablet_replica.cc:330] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: stopping tablet replica
I20260809 03:30:07.812656 18899 raft_consensus.cc:2229] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:07.812671 18900 tablet_replica.cc:330] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: stopping tablet replica
I20260809 03:30:07.812795 18047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:30:07.812963 18899 raft_consensus.cc:2258] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 03:30:07.813189 18047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:07.813201 18900 raft_consensus.cc:2229] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:07.813588 18900 raft_consensus.cc:2258] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:07.815120 18900 ts_tablet_manager.cc:1905] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:07.816329 18899 ts_tablet_manager.cc:1905] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:07.817430 18256 tablet_service.cc:1467] Processing CreateTablet for tablet 506160bbd2914b388e96a3fbeffe44ab (DEFAULT_TABLE table=test-table [id=7bf4d996e8c84337a9e39daaf73c389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.817690 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506160bbd2914b388e96a3fbeffe44ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.818131 18900 ts_tablet_manager.cc:1918] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:07.818228 18900 log.cc:1196] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/b8464015f8fd4f3890c596a3254bf18c
I20260809 03:30:07.818575 18900 ts_tablet_manager.cc:1939] T b8464015f8fd4f3890c596a3254bf18c P 3cee25a51c97455fb2a7047cddb10676: Deleting consensus metadata
I20260809 03:30:07.819589 18861 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Bootstrap starting.
I20260809 03:30:07.819972 18013 catalog_manager.cc:4841] TS 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417): tablet b8464015f8fd4f3890c596a3254bf18c (table test-table [id=4a46b31238474de2a7c36749caecdfea]) successfully deleted
I20260809 03:30:07.820271 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 506160bbd2914b388e96a3fbeffe44ab (DEFAULT_TABLE table=test-table [id=7bf4d996e8c84337a9e39daaf73c389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.820515 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506160bbd2914b388e96a3fbeffe44ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.820520 18861 tablet_bootstrap.cc:654] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.820590 18715 tablet_service.cc:1467] Processing CreateTablet for tablet 506160bbd2914b388e96a3fbeffe44ab (DEFAULT_TABLE table=test-table [id=7bf4d996e8c84337a9e39daaf73c389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:07.820815 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506160bbd2914b388e96a3fbeffe44ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:07.822616 18864 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Bootstrap starting.
I20260809 03:30:07.822715 18860 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Bootstrap starting.
I20260809 03:30:07.823498 18864 tablet_bootstrap.cc:654] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.823534 18860 tablet_bootstrap.cc:654] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:07.824671 18864 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: No bootstrap required, opened a new log
I20260809 03:30:07.824751 18864 ts_tablet_manager.cc:1397] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:07.825170 18864 raft_consensus.cc:348] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.825287 18864 raft_consensus.cc:374] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.825317 18864 raft_consensus.cc:729] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cee25a51c97455fb2a7047cddb10676, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.825433 18864 consensus_queue.cc:260] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.825663 18864 ts_tablet_manager.cc:1428] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:07.826010 18897 tablet_replica.cc:330] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: stopping tablet replica
I20260809 03:30:07.826303 18897 raft_consensus.cc:2229] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:07.826303 18899 ts_tablet_manager.cc:1918] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:07.826401 18899 log.cc:1196] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/b8464015f8fd4f3890c596a3254bf18c
I20260809 03:30:07.826608 18897 raft_consensus.cc:2258] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:07.827311 18899 ts_tablet_manager.cc:1939] T b8464015f8fd4f3890c596a3254bf18c P 24f0ad004bfb4540bcb67809e165cefa: Deleting consensus metadata
I20260809 03:30:07.828167 18897 ts_tablet_manager.cc:1905] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:07.828435 18012 catalog_manager.cc:4841] TS 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291): tablet b8464015f8fd4f3890c596a3254bf18c (table test-table [id=4a46b31238474de2a7c36749caecdfea]) successfully deleted
I20260809 03:30:07.829159 18860 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: No bootstrap required, opened a new log
I20260809 03:30:07.829238 18860 ts_tablet_manager.cc:1397] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260809 03:30:07.829664 18860 raft_consensus.cc:348] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.829800 18860 raft_consensus.cc:374] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.829839 18860 raft_consensus.cc:729] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5795897d1ca74405ac3083d9ad1af4d7, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.829955 18860 consensus_queue.cc:260] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.830202 18860 ts_tablet_manager.cc:1428] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:07.830654 18897 ts_tablet_manager.cc:1918] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:07.830737 18897 log.cc:1196] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/b8464015f8fd4f3890c596a3254bf18c
I20260809 03:30:07.831025 18897 ts_tablet_manager.cc:1939] T b8464015f8fd4f3890c596a3254bf18c P 5795897d1ca74405ac3083d9ad1af4d7: Deleting consensus metadata
I20260809 03:30:07.831583 18861 tablet_bootstrap.cc:492] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: No bootstrap required, opened a new log
I20260809 03:30:07.831664 18861 ts_tablet_manager.cc:1397] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260809 03:30:07.832067 18861 raft_consensus.cc:348] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.832180 18861 raft_consensus.cc:374] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:07.832221 18861 raft_consensus.cc:729] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f0ad004bfb4540bcb67809e165cefa, State: Initialized, Role: FOLLOWER
I20260809 03:30:07.832232 18013 catalog_manager.cc:4841] TS 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587): tablet b8464015f8fd4f3890c596a3254bf18c (table test-table [id=4a46b31238474de2a7c36749caecdfea]) successfully deleted
I20260809 03:30:07.832346 18861 consensus_queue.cc:260] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:07.832621 18861 ts_tablet_manager.cc:1428] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260809 03:30:07.851480 18598 tablet.cc:2369] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:08.014226 18873 raft_consensus.cc:482] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:08.014374 18873 raft_consensus.cc:504] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.015415 18873 leader_election.cc:288] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291), 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587)
I20260809 03:30:08.020077 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.016004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37791 (local address 127.15.222.66:42587)
0809 03:30:08.016145 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.016150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.016241 (+ 91us) server_negotiation.cc:407] Connection header received
0809 03:30:08.016317 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.016320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.016369 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.016443 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:08.016678 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.017405 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.018794 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.019409 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.019529 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.019582 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.019871 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.019944 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:30:08.020077 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.015799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:44215 (local address 127.15.222.65:35291)
0809 03:30:08.015934 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.015940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.015959 (+ 19us) server_negotiation.cc:407] Connection header received
0809 03:30:08.016078 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.016082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.016132 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.016208 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:08.016467 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.017254 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.018794 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.019403 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.019529 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.019582 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.019871 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.019952 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:08.020112 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.015688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35291 (local address 127.15.222.67:44215)
0809 03:30:08.015820 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.015837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.015958 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.016222 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.016225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.016239 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:08.016363 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.016368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.017378 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.017380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.018662 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.018674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.019534 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.019539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.019719 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.019773 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.019849 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.019963 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":56}
I20260809 03:30:08.020285 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.015689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42587 (local address 127.15.222.67:37791)
0809 03:30:08.016121 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.016133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.016236 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.016463 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.016466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.016483 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:08.016578 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.016582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.017499 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.017502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.018687 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.018695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.019546 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.019549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.019726 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.019766 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.019849 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.019978 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":178,"thread_start_us":80,"threads_started":1}
I20260809 03:30:08.020669 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506160bbd2914b388e96a3fbeffe44ab" candidate_uuid: "3cee25a51c97455fb2a7047cddb10676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f0ad004bfb4540bcb67809e165cefa" is_pre_election: true
I20260809 03:30:08.020743 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506160bbd2914b388e96a3fbeffe44ab" candidate_uuid: "3cee25a51c97455fb2a7047cddb10676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5795897d1ca74405ac3083d9ad1af4d7" is_pre_election: true
I20260809 03:30:08.020835 18290 raft_consensus.cc:2454] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cee25a51c97455fb2a7047cddb10676 in term 0.
I20260809 03:30:08.020970 18523 raft_consensus.cc:2454] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cee25a51c97455fb2a7047cddb10676 in term 0.
I20260809 03:30:08.021215 18639 leader_election.cc:302] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f0ad004bfb4540bcb67809e165cefa, 3cee25a51c97455fb2a7047cddb10676; no voters:
I20260809 03:30:08.021497 18873 raft_consensus.cc:2781] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:08.021551 18873 raft_consensus.cc:482] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:08.021577 18873 raft_consensus.cc:3037] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.022588 18873 raft_consensus.cc:504] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.022922 18873 leader_election.cc:288] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [CANDIDATE]: Term 1 election: Requested vote from peers 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291), 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587)
I20260809 03:30:08.023219 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506160bbd2914b388e96a3fbeffe44ab" candidate_uuid: "3cee25a51c97455fb2a7047cddb10676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f0ad004bfb4540bcb67809e165cefa"
I20260809 03:30:08.023255 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506160bbd2914b388e96a3fbeffe44ab" candidate_uuid: "3cee25a51c97455fb2a7047cddb10676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5795897d1ca74405ac3083d9ad1af4d7"
I20260809 03:30:08.023329 18290 raft_consensus.cc:3037] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.023366 18523 raft_consensus.cc:3037] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.024199 18290 raft_consensus.cc:2454] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cee25a51c97455fb2a7047cddb10676 in term 1.
I20260809 03:30:08.024314 18523 raft_consensus.cc:2454] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cee25a51c97455fb2a7047cddb10676 in term 1.
I20260809 03:30:08.024485 18639 leader_election.cc:302] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f0ad004bfb4540bcb67809e165cefa, 3cee25a51c97455fb2a7047cddb10676; no voters:
I20260809 03:30:08.024699 18873 raft_consensus.cc:2781] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:08.024911 18873 raft_consensus.cc:686] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 1 LEADER]: Becoming Leader. State: Replica: 3cee25a51c97455fb2a7047cddb10676, State: Running, Role: LEADER
I20260809 03:30:08.025159 18873 consensus_queue.cc:237] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.027361 18048 catalog_manager.cc:5495] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67). New cstate: current_term: 1 leader_uuid: "3cee25a51c97455fb2a7047cddb10676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } health_report { overall_health: HEALTHY } } }
W20260809 03:30:08.064987 18816 tablet.cc:2369] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:30:08 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246206, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:08.090838 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.083332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:36021 (local address 127.0.0.1:39268)
0809 03:30:08.083596 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.083609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.083711 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.083925 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.083927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.084311 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:08.084428 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.084434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.085094 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.085096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.085643 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.085650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.086059 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.086061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.086118 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.086602 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:08.086614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:08.087991 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:08.089534 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:08.089542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:08.089548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:08.089801 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:08.090013 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:08.090015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:08.090016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:08.090090 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:08.090373 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:08.090378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:08.090477 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.090562 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.090674 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260809 03:30:08.090853 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.083405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.15.222.126:36021)
0809 03:30:08.083536 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.083541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.083677 (+ 136us) server_negotiation.cc:407] Connection header received
0809 03:30:08.083783 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.083785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.083827 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.083902 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:08.084519 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.085019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.085751 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.085974 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.086051 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.088128 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:08.088132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:08.088135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:08.088157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:08.089410 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.089870 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.089873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.089875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.089949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.090148 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.090150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.090151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.090305 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:08.090372 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.090604 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.090730 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":52}
I20260809 03:30:08.094485 18048 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39268:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:30:08.095191 18048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:08.099661 18256 tablet_service.cc:1467] Processing CreateTablet for tablet 06657670a3554605a56272ecfb6a1812 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f92a7ae88d4b0b931fae72eef0270b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:08.099758 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 06657670a3554605a56272ecfb6a1812 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f92a7ae88d4b0b931fae72eef0270b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:08.100057 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06657670a3554605a56272ecfb6a1812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:08.100133 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06657670a3554605a56272ecfb6a1812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:08.102345 18860 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: Bootstrap starting.
I20260809 03:30:08.102420 18861 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: Bootstrap starting.
I20260809 03:30:08.102902 18715 tablet_service.cc:1467] Processing CreateTablet for tablet 06657670a3554605a56272ecfb6a1812 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f92a7ae88d4b0b931fae72eef0270b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:08.103247 18860 tablet_bootstrap.cc:654] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:08.103247 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06657670a3554605a56272ecfb6a1812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:08.103335 18861 tablet_bootstrap.cc:654] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:08.104221 18860 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: No bootstrap required, opened a new log
I20260809 03:30:08.104290 18860 ts_tablet_manager.cc:1397] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:08.104622 18860 raft_consensus.cc:348] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.104712 18860 raft_consensus.cc:374] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:08.104732 18860 raft_consensus.cc:729] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5795897d1ca74405ac3083d9ad1af4d7, State: Initialized, Role: FOLLOWER
I20260809 03:30:08.104838 18860 consensus_queue.cc:260] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.105087 18860 ts_tablet_manager.cc:1428] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:08.105053 18866 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.105170 18866 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.105871 18864 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: Bootstrap starting.
I20260809 03:30:08.106598 18864 tablet_bootstrap.cc:654] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:08.107717 18864 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: No bootstrap required, opened a new log
I20260809 03:30:08.107803 18864 ts_tablet_manager.cc:1397] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 03:30:08.108001 18861 tablet_bootstrap.cc:492] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: No bootstrap required, opened a new log
I20260809 03:30:08.108071 18861 ts_tablet_manager.cc:1397] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260809 03:30:08.108242 18864 raft_consensus.cc:348] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.108337 18864 raft_consensus.cc:374] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:08.108377 18864 raft_consensus.cc:729] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cee25a51c97455fb2a7047cddb10676, State: Initialized, Role: FOLLOWER
I20260809 03:30:08.108484 18861 raft_consensus.cc:348] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.108502 18864 consensus_queue.cc:260] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.108587 18861 raft_consensus.cc:374] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:08.108626 18861 raft_consensus.cc:729] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f0ad004bfb4540bcb67809e165cefa, State: Initialized, Role: FOLLOWER
I20260809 03:30:08.108754 18864 ts_tablet_manager.cc:1428] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:08.108740 18861 consensus_queue.cc:260] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.108942 18873 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.109057 18873 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.108984 18867 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.109126 18867 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.108997 18861 ts_tablet_manager.cc:1428] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:08.153198 18867 raft_consensus.cc:482] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:08.153311 18867 raft_consensus.cc:504] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.154448 18867 leader_election.cc:288] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587), 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417)
I20260809 03:30:08.158850 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.154619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42587 (local address 127.15.222.65:55017)
0809 03:30:08.154897 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.154912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.155065 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.155388 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.155391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.155405 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:08.155521 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.155526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.156405 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.156409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.157449 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.157459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.158322 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.158325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.158484 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.158514 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.158638 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.158711 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":31,"thread_start_us":108,"threads_started":1}
I20260809 03:30:08.158866 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.154727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:55017 (local address 127.15.222.66:42587)
0809 03:30:08.154853 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.154873 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.154987 (+ 114us) server_negotiation.cc:407] Connection header received
0809 03:30:08.155180 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.155183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.155227 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.155268 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:08.155640 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.156313 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.157571 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.158213 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.158323 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.158373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.158676 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.158747 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:08.159217 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.155110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42833 (local address 127.15.222.67:37417)
0809 03:30:08.155331 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.155335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.155560 (+ 225us) server_negotiation.cc:407] Connection header received
0809 03:30:08.155728 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.155731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.155799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.155862 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:08.156127 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.156864 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.158060 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.158544 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.158668 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.158715 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.159032 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.159108 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260809 03:30:08.159505 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06657670a3554605a56272ecfb6a1812" candidate_uuid: "24f0ad004bfb4540bcb67809e165cefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5795897d1ca74405ac3083d9ad1af4d7" is_pre_election: true
I20260809 03:30:08.159565 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.154999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37417 (local address 127.15.222.65:42833)
0809 03:30:08.155476 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.155487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.155579 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.155884 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.155887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.155916 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:08.156033 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.156039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.156988 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.156991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.157947 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.157956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.158785 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.158789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.158878 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.158912 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.159390 (+ 478us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.159463 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260809 03:30:08.159641 18523 raft_consensus.cc:2454] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24f0ad004bfb4540bcb67809e165cefa in term 0.
I20260809 03:30:08.159981 18185 leader_election.cc:302] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f0ad004bfb4540bcb67809e165cefa, 5795897d1ca74405ac3083d9ad1af4d7; no voters:
I20260809 03:30:08.160092 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06657670a3554605a56272ecfb6a1812" candidate_uuid: "24f0ad004bfb4540bcb67809e165cefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee25a51c97455fb2a7047cddb10676" is_pre_election: true
I20260809 03:30:08.160214 18867 raft_consensus.cc:2781] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:08.160228 18750 raft_consensus.cc:2454] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24f0ad004bfb4540bcb67809e165cefa in term 0.
I20260809 03:30:08.160266 18867 raft_consensus.cc:482] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:08.160300 18867 raft_consensus.cc:3037] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.161289 18867 raft_consensus.cc:504] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.161571 18867 leader_election.cc:288] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [CANDIDATE]: Term 1 election: Requested vote from peers 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587), 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417)
I20260809 03:30:08.161895 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06657670a3554605a56272ecfb6a1812" candidate_uuid: "24f0ad004bfb4540bcb67809e165cefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5795897d1ca74405ac3083d9ad1af4d7"
I20260809 03:30:08.161950 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06657670a3554605a56272ecfb6a1812" candidate_uuid: "24f0ad004bfb4540bcb67809e165cefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee25a51c97455fb2a7047cddb10676"
I20260809 03:30:08.162017 18523 raft_consensus.cc:3037] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.162065 18750 raft_consensus.cc:3037] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:08.162881 18523 raft_consensus.cc:2454] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24f0ad004bfb4540bcb67809e165cefa in term 1.
I20260809 03:30:08.163305 18750 raft_consensus.cc:2454] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24f0ad004bfb4540bcb67809e165cefa in term 1.
I20260809 03:30:08.163610 18185 leader_election.cc:302] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f0ad004bfb4540bcb67809e165cefa, 5795897d1ca74405ac3083d9ad1af4d7; no voters:
I20260809 03:30:08.163756 18867 raft_consensus.cc:2781] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:08.163935 18867 raft_consensus.cc:686] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [term 1 LEADER]: Becoming Leader. State: Replica: 24f0ad004bfb4540bcb67809e165cefa, State: Running, Role: LEADER
I20260809 03:30:08.164160 18867 consensus_queue.cc:237] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } }
I20260809 03:30:08.165280 18923 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: TxnStatusTablet state changed. Reason: New leader 24f0ad004bfb4540bcb67809e165cefa. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.165443 18923 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:08.165760 18925 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:30:08.165894 18925 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:30:08.166378 18047 catalog_manager.cc:5495] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa reported cstate change: term changed from 0 to 1, leader changed from <none> to 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65). New cstate: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:08.188020 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.184334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.15.222.65:35291)
0809 03:30:08.184443 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.184447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.184547 (+ 100us) server_negotiation.cc:407] Connection header received
0809 03:30:08.184716 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.184718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.184767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.184846 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:08.185082 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.185561 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.186748 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.186962 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.187035 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.187322 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.187694 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.187772 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.187861 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.187916 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:30:08.188083 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.184241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35291 (local address 127.0.0.1:40864)
0809 03:30:08.184478 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.184493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.184601 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.184866 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.184869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.184876 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:08.184995 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.185001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.185651 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.185654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.186648 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.186660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.187058 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.187061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.187153 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.187207 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.187774 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.187786 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.187853 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.187944 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260809 03:30:08.190845 18750 raft_consensus.cc:1261] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Refusing update from remote peer 24f0ad004bfb4540bcb67809e165cefa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:08.190959 18523 raft_consensus.cc:1261] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Refusing update from remote peer 24f0ad004bfb4540bcb67809e165cefa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:08.191288 18867 consensus_queue.cc:1035] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:08.191473 18923 consensus_queue.cc:1035] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:08.193135 18866 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: TxnStatusTablet state changed. Reason: New leader 24f0ad004bfb4540bcb67809e165cefa. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.193130 18873 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: TxnStatusTablet state changed. Reason: New leader 24f0ad004bfb4540bcb67809e165cefa. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.193255 18873 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.193266 18866 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.195089 18923 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.195242 18923 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:08.196520 18867 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.196621 18867 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 24f0ad004bfb4540bcb67809e165cefa: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:08.196846 18873 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.196961 18873 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 3cee25a51c97455fb2a7047cddb10676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.197359 18866 tablet_replica.cc:439] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24f0ad004bfb4540bcb67809e165cefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 } } peers { permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 } } peers { permanent_uuid: "3cee25a51c97455fb2a7047cddb10676" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37417 } } }
I20260809 03:30:08.197461 18866 tablet_replica.cc:442] T 06657670a3554605a56272ecfb6a1812 P 5795897d1ca74405ac3083d9ad1af4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:08.197901 18885 mvcc.cc:205] Tried to move back new op lower bound from 7316464468744675328 to 7316464468642770944. Current Snapshot: MvccSnapshot[applied={T|T < 7316464468744675328 or (T in {7316464468744675328})}]
I20260809 03:30:08.204775 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.201128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.15.222.67:37417)
0809 03:30:08.201248 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.201252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.201265 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:08.201430 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.201433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.201491 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.201578 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:08.201847 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.202597 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.203341 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.203679 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.203776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.203960 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.204443 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.204552 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.204581 (+ 29us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.204631 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:30:08.214699 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.211393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:33549 (local address 127.15.222.65:35291)
0809 03:30:08.211489 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.211492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.211722 (+ 230us) server_negotiation.cc:407] Connection header received
0809 03:30:08.211834 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.211836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.211872 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.211938 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:08.212158 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.212616 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.213800 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.214004 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.214072 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.214362 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.214419 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.214464 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.214566 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.214603 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:30:08.214776 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.211307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35291 (local address 127.15.222.67:33549)
0809 03:30:08.211648 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.211661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.211761 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.211959 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.211961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.211969 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:08.212079 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.212085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.212709 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.212713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.213672 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.213682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.214108 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.214111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.214205 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.214255 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.214489 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.214495 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.214551 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.214623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260809 03:30:08.221204 18523 raft_consensus.cc:1261] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Refusing update from remote peer 3cee25a51c97455fb2a7047cddb10676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:08.221294 18290 raft_consensus.cc:1261] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Refusing update from remote peer 3cee25a51c97455fb2a7047cddb10676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:08.221740 18906 consensus_queue.cc:1035] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f0ad004bfb4540bcb67809e165cefa" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:08.221915 18873 consensus_queue.cc:1035] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5795897d1ca74405ac3083d9ad1af4d7" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:08.225986 18886 mvcc.cc:205] Tried to move back new op lower bound from 7316464468870160384 to 7316464468073566208. Current Snapshot: MvccSnapshot[applied={T|T < 7316464468870160384}]
I20260809 03:30:08.243144 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.239906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:51915 (local address 127.15.222.67:37417)
0809 03:30:08.240133 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.240138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.240222 (+ 84us) server_negotiation.cc:407] Connection header received
0809 03:30:08.240357 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.240360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.240402 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.240460 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:08.240735 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.241231 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.242116 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.242334 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.242417 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.242674 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.242742 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.242810 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.242965 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.243020 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260809 03:30:08.243567 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.239845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37417 (local address 127.15.222.65:51915)
0809 03:30:08.240154 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:08.240168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:08.240269 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:08.240477 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:08.240479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:08.240487 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:08.240613 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.240620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.241340 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.241342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.242011 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:08.242021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.242433 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.242437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:08.242518 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.242559 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:08.242824 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:08.242831 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:30:08.242943 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 03:30:08.243021 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260809 03:30:08.266394 18048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:08.266577 18048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:08.268803 18048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e570e22f9fd43cbadd846b71bf981b1: Sending DeleteTablet for 3 replicas of tablet 506160bbd2914b388e96a3fbeffe44ab
I20260809 03:30:08.269819 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 506160bbd2914b388e96a3fbeffe44ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:08 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:56390
I20260809 03:30:08.270018 18897 tablet_replica.cc:330] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: stopping tablet replica
I20260809 03:30:08.270131 18897 raft_consensus.cc:2229] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:08.270237 18897 raft_consensus.cc:2258] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:08.270434 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet 506160bbd2914b388e96a3fbeffe44ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:08 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39216
I20260809 03:30:08.270588 18900 tablet_replica.cc:330] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: stopping tablet replica
I20260809 03:30:08.270684 18900 raft_consensus.cc:2229] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:08.270833 18256 tablet_service.cc:1514] Processing DeleteTablet for tablet 506160bbd2914b388e96a3fbeffe44ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:08 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:40852
I20260809 03:30:08.270975 18900 raft_consensus.cc:2258] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:08.271039 18899 tablet_replica.cc:330] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: stopping tablet replica
I20260809 03:30:08.271153 18899 raft_consensus.cc:2229] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:08.271271 18899 raft_consensus.cc:2258] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:08.271832 18897 ts_tablet_manager.cc:1905] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:08.272022 18899 ts_tablet_manager.cc:1905] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:08.274613 18899 ts_tablet_manager.cc:1918] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:08.274696 18899 log.cc:1196] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/506160bbd2914b388e96a3fbeffe44ab
I20260809 03:30:08.274686 18897 ts_tablet_manager.cc:1918] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:08.274748 18897 log.cc:1196] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/506160bbd2914b388e96a3fbeffe44ab
I20260809 03:30:08.274948 18899 ts_tablet_manager.cc:1939] T 506160bbd2914b388e96a3fbeffe44ab P 24f0ad004bfb4540bcb67809e165cefa: Deleting consensus metadata
I20260809 03:30:08.274984 18897 ts_tablet_manager.cc:1939] T 506160bbd2914b388e96a3fbeffe44ab P 5795897d1ca74405ac3083d9ad1af4d7: Deleting consensus metadata
I20260809 03:30:08.275590 18013 catalog_manager.cc:4841] TS 5795897d1ca74405ac3083d9ad1af4d7 (127.15.222.66:42587): tablet 506160bbd2914b388e96a3fbeffe44ab (table test-table [id=7bf4d996e8c84337a9e39daaf73c389d]) successfully deleted
I20260809 03:30:08.275590 18012 catalog_manager.cc:4841] TS 24f0ad004bfb4540bcb67809e165cefa (127.15.222.65:35291): tablet 506160bbd2914b388e96a3fbeffe44ab (table test-table [id=7bf4d996e8c84337a9e39daaf73c389d]) successfully deleted
I20260809 03:30:08.276299 18900 ts_tablet_manager.cc:1905] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:08.279048 18900 ts_tablet_manager.cc:1918] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:08.279126 18900 log.cc:1196] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/506160bbd2914b388e96a3fbeffe44ab
I20260809 03:30:08.279358 18900 ts_tablet_manager.cc:1939] T 506160bbd2914b388e96a3fbeffe44ab P 3cee25a51c97455fb2a7047cddb10676: Deleting consensus metadata
I20260809 03:30:08.280032 18013 catalog_manager.cc:4841] TS 3cee25a51c97455fb2a7047cddb10676 (127.15.222.67:37417): tablet 506160bbd2914b388e96a3fbeffe44ab (table test-table [id=7bf4d996e8c84337a9e39daaf73c389d]) successfully deleted
Aug 09 03:30:08 dist-test-slave-cvwc krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:08.286190 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.271485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.15.222.65:35291)
0809 03:30:08.271592 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.271596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.276291 (+ 4695us) server_negotiation.cc:407] Connection header received
0809 03:30:08.276499 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.276502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.276548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.276598 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:08.277526 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.278263 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.279434 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.279720 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.279796 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.282957 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:08.282968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:08.282980 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:08.283002 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:08.284491 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.284876 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.284881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.284883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.284983 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.285233 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.285236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.285238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.285466 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:08.285661 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.285884 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.286003 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 03:30:08.286621 18332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40866
I20260809 03:30:08.295481 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:08.288081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.15.222.126:36021)
0809 03:30:08.288203 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:08.288207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:08.288567 (+ 360us) server_negotiation.cc:407] Connection header received
0809 03:30:08.288697 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:08.288700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:08.288746 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:08.288808 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:08.289431 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.290093 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.290824 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:08.291048 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:08.291134 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:08.292291 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:08.292295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:08.292298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:08.292323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:08.293635 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.294239 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.294243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.294244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.294334 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:08.294619 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:08.294622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:08.294624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:08.294818 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:08.294931 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:08.295172 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 03:30:08.295317 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 03:30:08.295969 18048 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39280
I20260809 03:30:08.298322 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18159
I20260809 03:30:08.313086 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18379
I20260809 03:30:08.328483 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18611
I20260809 03:30:08.339035 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17982
2026-08-09T03:30:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2894 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:08 dist-test-slave-cvwc krb5kdc[18957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:08 dist-test-slave-cvwc krb5kdc[18957](info): set up 2 sockets
Aug 09 03:30:08 dist-test-slave-cvwc krb5kdc[18957](info): commencing operation
krb5kdc: starting...
W20260809 03:30:09.494115 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:09 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:09.610980 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:34335
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:09.722018 18978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:09.722265 18978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:09.722317 18978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:09.726239 18978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:09.726292 18978 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:30:09.726315 18978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:09.726341 18978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:09.726369 18978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:09.726392 18978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:09.731588 18978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:34335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:09.732564 18978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:09.733546 18978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:09.741557 18994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:09.741547 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:09.741549 18986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:09.741916 18978 server_base.cc:1034] running on GCE node
I20260809 03:30:09.742424 18978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:09.743419 18978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:09.744567 18978 hybrid_clock.cc:648] HybridClock initialized: now 1786246209744547 us; error 33 us; skew 500 ppm
Aug 09 03:30:09 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246209, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:09.747190 18978 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:09.747995 18978 webserver.cc:460] Webserver started at http://127.15.222.126:44831/ using document root <none> and password file <none>
I20260809 03:30:09.748490 18978 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:09.748546 18978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:09.748704 18978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:09.750670 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "2def972f29ad44a4ae66fec0b1062ec2"
format_stamp: "Formatted at 2026-08-09 03:30:09 on dist-test-slave-cvwc"
server_key: "407992f67c4c38d2c68ea4ec3735cba4"
server_key_iv: "5a872c6342b45b9a0858329f08686804"
server_key_version: "encryptionkey@0"
I20260809 03:30:09.751166 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "2def972f29ad44a4ae66fec0b1062ec2"
format_stamp: "Formatted at 2026-08-09 03:30:09 on dist-test-slave-cvwc"
server_key: "407992f67c4c38d2c68ea4ec3735cba4"
server_key_iv: "5a872c6342b45b9a0858329f08686804"
server_key_version: "encryptionkey@0"
I20260809 03:30:09.754739 18978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:09.756994 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:09.758164 18978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:09.758277 18978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "2def972f29ad44a4ae66fec0b1062ec2"
format_stamp: "Formatted at 2026-08-09 03:30:09 on dist-test-slave-cvwc"
server_key: "407992f67c4c38d2c68ea4ec3735cba4"
server_key_iv: "5a872c6342b45b9a0858329f08686804"
server_key_version: "encryptionkey@0"
I20260809 03:30:09.758370 18978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:09.778638 18978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:09.781816 18978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:09.782006 18978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:09.789291 18978 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:34335
I20260809 03:30:09.789323 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:34335 every 8 connection(s)
I20260809 03:30:09.790225 18978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:09.792918 19112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:09.797166 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18978
I20260809 03:30:09.797271 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:09.797554 16249 external_mini_cluster.cc:1428] Setting key 6a53b8dc566612f8eca48ec61d1fe18e
I20260809 03:30:09.798604 19112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Bootstrap starting.
I20260809 03:30:09.800743 19112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:09.801635 19112 log.cc:824] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:09.803647 19112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: No bootstrap required, opened a new log
Aug 09 03:30:09 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:09.807523 19112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } }
I20260809 03:30:09.807789 19112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:09.807850 19112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2def972f29ad44a4ae66fec0b1062ec2, State: Initialized, Role: FOLLOWER
I20260809 03:30:09.808328 19112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } }
I20260809 03:30:09.808493 19112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:09.808560 19112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:09.808660 19112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:09.809607 19112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } }
I20260809 03:30:09.809988 19112 leader_election.cc:302] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2def972f29ad44a4ae66fec0b1062ec2; no voters:
I20260809 03:30:09.810331 19112 leader_election.cc:288] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:09.810461 19117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:09.810720 19117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 2def972f29ad44a4ae66fec0b1062ec2, State: Running, Role: LEADER
I20260809 03:30:09.811040 19117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } }
I20260809 03:30:09.811610 19112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:09.812696 19119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2def972f29ad44a4ae66fec0b1062ec2. Latest consensus state: current_term: 1 leader_uuid: "2def972f29ad44a4ae66fec0b1062ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } } }
I20260809 03:30:09.812827 19119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:09.812860 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:09.799032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.15.222.126:34335)
0809 03:30:09.799462 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:09.799478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:09.799515 (+ 37us) server_negotiation.cc:407] Connection header received
0809 03:30:09.800137 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:09.800154 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:09.800475 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:09.800756 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:09.801602 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:09.802827 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:09.803927 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:09.804385 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:09.804515 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:09.807189 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:09.807195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:09.807210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:09.807240 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:09.809196 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:09.809637 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:09.809643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:09.809649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:09.809815 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:09.810099 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:09.810102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:09.810104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:09.810704 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:09.811465 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:09.811612 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 03:30:09.811898 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260809 03:30:09.813138 19118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2def972f29ad44a4ae66fec0b1062ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def972f29ad44a4ae66fec0b1062ec2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 34335 } } }
I20260809 03:30:09.813238 19118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2 [sys.catalog]: This master's current role is: LEADER
W20260809 03:30:09.816309 19140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:09.816382 19140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:09.816444 19129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:09.817150 19129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:09.823055 19129 catalog_manager.cc:1347] Generated new cluster ID: 80b6f47aa5484d21b6e4eb81479a3584
I20260809 03:30:09.823119 19129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:09.865401 19129 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:09.866444 19129 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:09.873484 19129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Generated new TSK 0
I20260809 03:30:09.874186 19129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:09.910413 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34335
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:10.020041 19150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:10.020293 19150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:10.020358 19150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:10.024307 19150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:10.024384 19150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:10.024444 19150 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:30:10.024469 19150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:10.024514 19150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:10.029685 19150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:10.030681 19150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:10.031651 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:10.040768 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:10.040797 19156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:10.040781 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:10.041101 19150 server_base.cc:1034] running on GCE node
I20260809 03:30:10.041481 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:10.042083 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:10.043223 19150 hybrid_clock.cc:648] HybridClock initialized: now 1786246210043200 us; error 37 us; skew 500 ppm
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:10.045948 19150 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:10.046749 19150 webserver.cc:460] Webserver started at http://127.15.222.65:46769/ using document root <none> and password file <none>
I20260809 03:30:10.047255 19150 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:10.047312 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:10.047472 19150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:10.049386 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "7e62ef8993744390890b01c7987d17d6"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "fb0bfb8b69de39ffe61d1a6d289a49c2"
server_key_iv: "d24e2635d7ec35c6dad7c811599b987a"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.049935 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "7e62ef8993744390890b01c7987d17d6"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "fb0bfb8b69de39ffe61d1a6d289a49c2"
server_key_iv: "d24e2635d7ec35c6dad7c811599b987a"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.053395 19150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 03:30:10.055691 19173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.056838 19150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:10.056953 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "7e62ef8993744390890b01c7987d17d6"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "fb0bfb8b69de39ffe61d1a6d289a49c2"
server_key_iv: "d24e2635d7ec35c6dad7c811599b987a"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.057044 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:10.073570 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:10.076651 19150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:10.076818 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:10.078284 19150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:10.078332 19150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.078385 19150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:10.078415 19150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:10.088313 19150 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:37615
I20260809 03:30:10.088378 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:37615 every 8 connection(s)
I20260809 03:30:10.089311 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:10.091733 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.078929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:37453 (local address 127.15.222.126:34335)
0809 03:30:10.079079 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.079084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.080354 (+ 1270us) server_negotiation.cc:407] Connection header received
0809 03:30:10.081402 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.081406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.081462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.081535 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.082987 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.083699 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.084964 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.085310 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.085403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.088204 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.088208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.088211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.088241 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.089747 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.090292 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.090295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.090297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.090378 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.090685 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.090689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.090690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.090872 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.090959 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.091431 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.091571 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:10.092422 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.079726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.65:37453)
0809 03:30:10.080209 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.080258 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.081197 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.081734 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.081742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.082178 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.082831 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.082851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.083816 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.083819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.084838 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.084849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.085471 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.085475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.085651 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.086285 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.086306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.088035 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.089873 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.089879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.089906 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.090196 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.090466 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.090469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.090471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.090600 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.090971 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.090977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.091202 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.091361 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.091657 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":149,"threads_started":1}
I20260809 03:30:10.093585 19367 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34335
I20260809 03:30:10.093739 19367 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:10.094211 19367 heartbeater.cc:511] Master 127.15.222.126:34335 requested a full tablet report, sending...
I20260809 03:30:10.095767 19031 ts_manager.cc:194] Registered new tserver with Master: 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615)
I20260809 03:30:10.096702 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19150
I20260809 03:30:10.096819 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:10.097102 16249 external_mini_cluster.cc:1428] Setting key d121d1a143f413d5cc37304702b063e8
I20260809 03:30:10.097152 19031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:37453
I20260809 03:30:10.102752 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.095625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47719 (local address 127.15.222.126:34335)
0809 03:30:10.095723 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.095726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.095915 (+ 189us) server_negotiation.cc:407] Connection header received
0809 03:30:10.096023 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.096025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.096062 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.096125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.096595 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.097302 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.098382 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.098707 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.098812 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.099882 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.099885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.099887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.099915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.101276 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.101651 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.101654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.101656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.101765 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.102019 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.102022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.102024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.102183 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.102266 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.102511 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.102610 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:30:10.102762 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.095554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.65:47719)
0809 03:30:10.095843 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.095857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.095952 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.096156 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.096159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.096344 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.096474 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.096482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.097433 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.097437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.098245 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.098254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.098831 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.098834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.098914 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.099366 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.099379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.099755 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.101373 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.101376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.101377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.101564 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.101848 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.101851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.101853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.101937 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.102272 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.102275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.102359 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.102449 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.102580 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:10.122388 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34335
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:10.232956 19385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:10.233210 19385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:10.233274 19385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:10.237406 19385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:10.237483 19385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:10.237536 19385 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:30:10.237563 19385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:10.237604 19385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:10.242888 19385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:10.243804 19385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:10.244727 19385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:10.253803 19398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:10.253798 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:10.253798 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:10.254146 19385 server_base.cc:1034] running on GCE node
I20260809 03:30:10.254518 19385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:10.255122 19385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:10.256292 19385 hybrid_clock.cc:648] HybridClock initialized: now 1786246210256272 us; error 34 us; skew 500 ppm
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:10.258975 19385 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:10.259826 19385 webserver.cc:460] Webserver started at http://127.15.222.66:42777/ using document root <none> and password file <none>
I20260809 03:30:10.260345 19385 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:10.260411 19385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:10.260607 19385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:10.262701 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "684aff7e341542edb74eea43e068c195"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "942addfc55644521eaa4a6e9b2cfedd3"
server_key_iv: "8f73b7ab9d91560ad76cf265de1dcbe5"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.263248 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "684aff7e341542edb74eea43e068c195"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "942addfc55644521eaa4a6e9b2cfedd3"
server_key_iv: "8f73b7ab9d91560ad76cf265de1dcbe5"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.266896 19385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:10.269311 19410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.270583 19385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:10.270691 19385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "684aff7e341542edb74eea43e068c195"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "942addfc55644521eaa4a6e9b2cfedd3"
server_key_iv: "8f73b7ab9d91560ad76cf265de1dcbe5"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.270783 19385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:10.291787 19385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:10.294557 19385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:10.294767 19385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:10.296255 19385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:10.296309 19385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.296367 19385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:10.296399 19385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:10.308192 19385 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:45841
I20260809 03:30:10.308231 19592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:45841 every 8 connection(s)
I20260809 03:30:10.309077 19385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:10.309669 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.296859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:53077 (local address 127.15.222.126:34335)
0809 03:30:10.297011 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.297017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.298384 (+ 1367us) server_negotiation.cc:407] Connection header received
0809 03:30:10.299608 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.299612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.299669 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.299758 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.301004 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.301524 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.302731 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.303078 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.303215 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.305855 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.305860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.305864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.305899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.307532 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.308158 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.308162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.308164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.308267 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.308621 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.308625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.308627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.308812 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.308907 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.309379 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.309509 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:30:10.310434 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.297669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.66:53077)
0809 03:30:10.298195 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.298267 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.299364 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.299929 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.299938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.300285 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.300858 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.300873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.301647 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.301650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.302610 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.302620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.303256 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.303260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.303444 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.303954 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.303969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.305665 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.307645 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.307651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.307666 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.308028 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.308378 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.308381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.308383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.308518 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.308911 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.308917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.309148 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.309300 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.309605 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":110,"threads_started":1}
I20260809 03:30:10.311571 19594 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34335
I20260809 03:30:10.311757 19594 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:10.312282 19594 heartbeater.cc:511] Master 127.15.222.126:34335 requested a full tablet report, sending...
I20260809 03:30:10.313383 19031 ts_manager.cc:194] Registered new tserver with Master: 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841)
I20260809 03:30:10.314203 19031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:53077
I20260809 03:30:10.317757 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19385
I20260809 03:30:10.317873 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:10.318154 16249 external_mini_cluster.cc:1428] Setting key be00f7d67f4e6f0bc08e8cc398e5c7f9
I20260809 03:30:10.319875 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.313541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34139 (local address 127.15.222.126:34335)
0809 03:30:10.313634 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.313639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.313844 (+ 205us) server_negotiation.cc:407] Connection header received
0809 03:30:10.313953 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.313956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.313995 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.314057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.314478 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.315047 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.315827 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.316045 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.316126 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.317042 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.317045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.317048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.317070 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.318350 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.318788 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.318792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.318794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.318869 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.319108 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.319111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.319113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.319307 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.319400 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.319630 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.319728 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":24}
I20260809 03:30:10.319881 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.313475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.66:34139)
0809 03:30:10.313771 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.313785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.313871 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.314081 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.314083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.314261 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.314378 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.314384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.315168 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.315172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.315725 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.315733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.316139 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.316142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.316233 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.316614 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.316623 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.316913 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.318447 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.318450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.318452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.318697 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.318949 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.318952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.318954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.319032 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.319396 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.319399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.319485 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.319569 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.319697 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:10.343878 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:34335
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:10.452217 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:10.452453 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:10.452503 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:10.456439 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:10.456489 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:10.456537 19609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 03:30:10.456563 19609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:10.456615 19609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:10.461736 19609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:10.462702 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:10.463680 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:10.472711 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:10.472718 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:10.472808 19609 server_base.cc:1034] running on GCE node
W20260809 03:30:10.472718 19618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:10.473376 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:10.474007 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:10.475163 19609 hybrid_clock.cc:648] HybridClock initialized: now 1786246210475145 us; error 30 us; skew 500 ppm
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:10.477787 19609 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:10.478636 19609 webserver.cc:460] Webserver started at http://127.15.222.67:41857/ using document root <none> and password file <none>
I20260809 03:30:10.479159 19609 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:10.479224 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:10.479429 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:10.481400 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "723ea1f68a014927a8a4c7fbc894c59d"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "8962a45980c721fe81b5d343e24349a2"
server_key_iv: "aee9f546e5ab0b32d1029e11793ab422"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.481943 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "723ea1f68a014927a8a4c7fbc894c59d"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "8962a45980c721fe81b5d343e24349a2"
server_key_iv: "aee9f546e5ab0b32d1029e11793ab422"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.485383 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 03:30:10.487661 19630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.488799 19609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:10.488904 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "723ea1f68a014927a8a4c7fbc894c59d"
format_stamp: "Formatted at 2026-08-09 03:30:10 on dist-test-slave-cvwc"
server_key: "8962a45980c721fe81b5d343e24349a2"
server_key_iv: "aee9f546e5ab0b32d1029e11793ab422"
server_key_version: "encryptionkey@0"
I20260809 03:30:10.488996 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:10.510461 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:10.513293 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:10.513444 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:10.514817 19609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:10.514866 19609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:10.514916 19609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:10.514945 19609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:10.524355 19609 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:39305
I20260809 03:30:10.524415 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:39305 every 8 connection(s)
I20260809 03:30:10.525333 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:10.527988 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.515563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:33983 (local address 127.15.222.126:34335)
0809 03:30:10.515715 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.515720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.516510 (+ 790us) server_negotiation.cc:407] Connection header received
0809 03:30:10.517589 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.517593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.517655 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.517837 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.519258 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.520064 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.521674 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.522034 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.522151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.524786 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.524791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.524793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.524822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.526170 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.526730 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.526734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.526735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.526814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.527121 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.527123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.527124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.527290 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.527361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.527763 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.527859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:10.528764 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.515911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.67:33983)
0809 03:30:10.516352 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.516400 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.517350 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.518026 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.518036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.518423 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.519080 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.519101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.520192 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.520197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.521549 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.521560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.522173 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.522176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.522314 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.522986 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.523007 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.524634 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.526283 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.526288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.526301 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.526614 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.526906 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.526908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.526910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.527050 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.527364 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.527369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.527586 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.527727 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.527953 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260809 03:30:10.529203 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19609
I20260809 03:30:10.529299 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:10.529592 16249 external_mini_cluster.cc:1428] Setting key a3488e73aaed0bd4ab9ff969c8696388
I20260809 03:30:10.530084 19830 heartbeater.cc:347] Connected to a master server at 127.15.222.126:34335
I20260809 03:30:10.530331 19830 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:10.530865 19830 heartbeater.cc:511] Master 127.15.222.126:34335 requested a full tablet report, sending...
I20260809 03:30:10.532452 19033 ts_manager.cc:194] Registered new tserver with Master: 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:10.533136 19033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:33983
I20260809 03:30:10.537890 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.532076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:42399 (local address 127.15.222.126:34335)
0809 03:30:10.532183 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.532187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.532316 (+ 129us) server_negotiation.cc:407] Connection header received
0809 03:30:10.532422 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.532424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.532459 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.532517 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.532950 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.533519 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.534348 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.534642 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.534745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.535603 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.535605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.535607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.535629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.536664 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.537007 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.537009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.537010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.537074 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.537263 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.537264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.537266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.537414 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.537481 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.537648 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.537778 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:30:10.537899 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.531993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.15.222.67:42399)
0809 03:30:10.532250 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.532263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.532352 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.532550 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.532553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.532729 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.532836 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.532843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.533627 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.533630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.534237 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.534245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.534747 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.534749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.534817 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.535200 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.535211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.535490 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.536750 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.536753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.536754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.536937 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.537136 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.537138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.537139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.537204 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.537477 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.537479 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.537545 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.537615 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.537704 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260809 03:30:10.542902 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:10.566946 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.559791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.15.222.126:34335)
0809 03:30:10.559916 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.559919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.559931 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:10.559964 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.559966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.560000 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.560058 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.560746 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.561277 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.562130 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.562334 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.562416 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.564484 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.564486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.564489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.564512 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.565782 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.566107 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.566109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.566110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.566177 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.566349 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.566351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.566352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.566510 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.566601 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.566750 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.566833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:10.569533 19033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:10.572540 19033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:10.586236 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.580694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.15.222.66:45841)
0809 03:30:10.581583 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.581592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.581611 (+ 19us) server_negotiation.cc:407] Connection header received
0809 03:30:10.581678 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.581683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.581927 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.582077 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.582349 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.583259 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.584682 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.585469 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.585579 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.585763 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.586015 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.586103 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":115,"threads_started":1}
I20260809 03:30:10.586228 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.580503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:45841 (local address 127.0.0.1:42638)
0809 03:30:10.580848 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.580898 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.581018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.582065 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.582071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.582099 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.582244 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.582251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.583373 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.583376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.584561 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.584571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.585591 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.585595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.585706 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.585870 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.585962 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.586081 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":60,"spinlock_wait_cycles":3200,"thread_start_us":91,"threads_started":1}
I20260809 03:30:10.586449 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.581230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:39305 (local address 127.0.0.1:47504)
0809 03:30:10.581994 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.582006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.582099 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.582566 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.582569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.582585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.582679 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.582686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.583775 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.583778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.584860 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.584869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.586061 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.586065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.586150 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.586174 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.586239 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.586300 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
I20260809 03:30:10.586817 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.581526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.15.222.67:39305)
0809 03:30:10.581860 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.581870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.582193 (+ 323us) server_negotiation.cc:407] Connection header received
0809 03:30:10.582278 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.582283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.582453 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.582572 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.582801 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.583676 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.584974 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.585753 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.585846 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.585998 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.586597 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.586684 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260809 03:30:10.588092 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.580894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.0.0.1:48192)
0809 03:30:10.581249 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.581262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.581351 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.583166 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.583168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.583184 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.583261 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.583267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.585625 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.585627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.586767 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.586777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.587757 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.587760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.587845 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.587868 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.587915 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.587974 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":22,"thread_start_us":97,"threads_started":1}
I20260809 03:30:10.588685 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.581314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.15.222.65:37615)
0809 03:30:10.582829 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.582842 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.582859 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:30:10.582928 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.582931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.583054 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.583169 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.584711 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.585504 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.586891 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.587622 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.588264 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.588422 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.588470 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.588552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":76,"threads_started":1}
I20260809 03:30:10.589252 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 8b04c27b9b9743748120deba81a8ff05 (DEFAULT_TABLE table=test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:10.589226 19494 tablet_service.cc:1467] Processing CreateTablet for tablet 8b04c27b9b9743748120deba81a8ff05 (DEFAULT_TABLE table=test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:10.590296 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b04c27b9b9743748120deba81a8ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.590306 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b04c27b9b9743748120deba81a8ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.591202 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 8b04c27b9b9743748120deba81a8ff05 (DEFAULT_TABLE table=test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:10.592083 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b04c27b9b9743748120deba81a8ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.597016 19863 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Bootstrap starting.
I20260809 03:30:10.598282 19865 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Bootstrap starting.
I20260809 03:30:10.599152 19863 tablet_bootstrap.cc:654] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.599198 19866 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Bootstrap starting.
I20260809 03:30:10.599848 19863 log.cc:824] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:10.600351 19865 tablet_bootstrap.cc:654] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.601155 19865 log.cc:824] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:10.601528 19866 tablet_bootstrap.cc:654] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.601688 19863 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: No bootstrap required, opened a new log
I20260809 03:30:10.601907 19863 ts_tablet_manager.cc:1397] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:10.602306 19866 log.cc:824] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:10.602998 19865 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: No bootstrap required, opened a new log
I20260809 03:30:10.603170 19865 ts_tablet_manager.cc:1397] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:30:10.604193 19866 tablet_bootstrap.cc:492] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: No bootstrap required, opened a new log
I20260809 03:30:10.604363 19866 ts_tablet_manager.cc:1397] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:10.605041 19863 raft_consensus.cc:348] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.605268 19863 raft_consensus.cc:374] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.605317 19863 raft_consensus.cc:729] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.605752 19863 consensus_queue.cc:260] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.606353 19830 heartbeater.cc:503] Master 127.15.222.126:34335 was elected leader, sending a full tablet report...
I20260809 03:30:10.606714 19863 ts_tablet_manager.cc:1428] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:10.606825 19865 raft_consensus.cc:348] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.607067 19865 raft_consensus.cc:374] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.607112 19865 raft_consensus.cc:729] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684aff7e341542edb74eea43e068c195, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.607525 19865 consensus_queue.cc:260] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.607893 19866 raft_consensus.cc:348] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.608129 19866 raft_consensus.cc:374] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.608182 19594 heartbeater.cc:503] Master 127.15.222.126:34335 was elected leader, sending a full tablet report...
I20260809 03:30:10.608191 19866 raft_consensus.cc:729] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.608608 19866 consensus_queue.cc:260] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.609426 19367 heartbeater.cc:503] Master 127.15.222.126:34335 was elected leader, sending a full tablet report...
I20260809 03:30:10.609489 19865 ts_tablet_manager.cc:1428] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:30:10.609489 19866 ts_tablet_manager.cc:1428] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:30:10.742501 19874 raft_consensus.cc:482] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:10.742715 19874 raft_consensus.cc:504] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.743876 19874 leader_election.cc:288] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615)
I20260809 03:30:10.747316 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.744106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:45841 (local address 127.15.222.67:39061)
0809 03:30:10.744259 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.744277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.744369 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.744646 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.744648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.744666 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.744772 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.744777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.745487 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.745490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.746281 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.746290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.746920 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.746924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.747030 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.747048 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.747114 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.747184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 03:30:10.747316 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:39061 (local address 127.15.222.66:45841)
0809 03:30:10.744298 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.744302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.744319 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:30:10.744511 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.744514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.744559 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.744619 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.744871 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.745379 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.746404 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.746806 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.746921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.746979 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.747139 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.747216 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:30:10.747781 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.744124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.15.222.67:60049)
0809 03:30:10.744515 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.744526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.744626 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.744843 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.744846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.744861 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.745304 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.745311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.746072 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.746074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.746782 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.746790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.747383 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.747387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.747484 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.747506 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.747605 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.747667 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":155,"threads_started":1}
I20260809 03:30:10.747833 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.744334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:60049 (local address 127.15.222.65:37615)
0809 03:30:10.744464 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.744468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.744596 (+ 128us) server_negotiation.cc:407] Connection header received
0809 03:30:10.744722 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.744725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.744765 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.744826 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.745425 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.745964 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.746894 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.747262 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.747364 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.747423 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.747646 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.747724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 03:30:10.747984 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b04c27b9b9743748120deba81a8ff05" candidate_uuid: "723ea1f68a014927a8a4c7fbc894c59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195" is_pre_election: true
I20260809 03:30:10.748291 19522 raft_consensus.cc:2454] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723ea1f68a014927a8a4c7fbc894c59d in term 0.
I20260809 03:30:10.748391 19294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b04c27b9b9743748120deba81a8ff05" candidate_uuid: "723ea1f68a014927a8a4c7fbc894c59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e62ef8993744390890b01c7987d17d6" is_pre_election: true
I20260809 03:30:10.748674 19294 raft_consensus.cc:2454] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723ea1f68a014927a8a4c7fbc894c59d in term 0.
I20260809 03:30:10.748777 19636 leader_election.cc:302] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 723ea1f68a014927a8a4c7fbc894c59d; no voters:
I20260809 03:30:10.749055 19874 raft_consensus.cc:2781] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:10.749156 19874 raft_consensus.cc:482] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:10.749200 19874 raft_consensus.cc:3037] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.750255 19874 raft_consensus.cc:504] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.750558 19874 leader_election.cc:288] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [CANDIDATE]: Term 1 election: Requested vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615)
I20260809 03:30:10.750888 19294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b04c27b9b9743748120deba81a8ff05" candidate_uuid: "723ea1f68a014927a8a4c7fbc894c59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e62ef8993744390890b01c7987d17d6"
I20260809 03:30:10.750888 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b04c27b9b9743748120deba81a8ff05" candidate_uuid: "723ea1f68a014927a8a4c7fbc894c59d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195"
I20260809 03:30:10.751015 19522 raft_consensus.cc:3037] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.751015 19294 raft_consensus.cc:3037] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.752256 19294 raft_consensus.cc:2454] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723ea1f68a014927a8a4c7fbc894c59d in term 1.
I20260809 03:30:10.752256 19522 raft_consensus.cc:2454] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723ea1f68a014927a8a4c7fbc894c59d in term 1.
I20260809 03:30:10.752552 19636 leader_election.cc:302] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 723ea1f68a014927a8a4c7fbc894c59d; no voters:
I20260809 03:30:10.752739 19874 raft_consensus.cc:2781] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:10.752954 19874 raft_consensus.cc:686] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 LEADER]: Becoming Leader. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Running, Role: LEADER
I20260809 03:30:10.753250 19874 consensus_queue.cc:237] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } }
I20260809 03:30:10.756321 19030 catalog_manager.cc:5495] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d reported cstate change: term changed from 0 to 1, leader changed from <none> to 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67). New cstate: current_term: 1 leader_uuid: "723ea1f68a014927a8a4c7fbc894c59d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } health_report { overall_health: UNKNOWN } } }
W20260809 03:30:10.776268 19831 tablet.cc:2369] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:30:10 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246209, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:10.795435 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.787407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.15.222.126:34335)
0809 03:30:10.787531 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.787535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.787735 (+ 200us) server_negotiation.cc:407] Connection header received
0809 03:30:10.787843 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.787845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.787890 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.787954 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:10.788541 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.789070 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.789850 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.790056 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.790137 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.792454 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:10.792457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:10.792458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:10.792478 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:10.793875 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.794272 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.794275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.794277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.794360 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:10.794597 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:10.794600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:10.794602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:10.794767 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:10.794846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.795161 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.795303 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":44}
I20260809 03:30:10.795435 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.787358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:34335 (local address 127.0.0.1:47478)
0809 03:30:10.787653 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.787667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.787776 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.787980 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.787982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.788360 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:10.788455 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.788461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.789169 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.789172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.789745 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.789755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.790154 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.790158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.790231 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.790756 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:10.790768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:10.792329 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:10.793965 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.793975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.793980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.794188 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.794441 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:10.794444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:10.794446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:10.794524 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:10.794901 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:10.794908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:10.795008 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.795102 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.795251 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
I20260809 03:30:10.799566 19030 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:30:10.800235 19030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:10.804734 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 175ce4f74ee94234bc8d049d12440e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e45d76b007384065956f10aaeb59a5e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:10.804891 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 175ce4f74ee94234bc8d049d12440e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e45d76b007384065956f10aaeb59a5e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:10.804852 19494 tablet_service.cc:1467] Processing CreateTablet for tablet 175ce4f74ee94234bc8d049d12440e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e45d76b007384065956f10aaeb59a5e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:10.805446 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ce4f74ee94234bc8d049d12440e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.805446 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ce4f74ee94234bc8d049d12440e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.805552 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175ce4f74ee94234bc8d049d12440e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:10.808303 19866 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: Bootstrap starting.
I20260809 03:30:10.808385 19865 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: Bootstrap starting.
I20260809 03:30:10.809181 19866 tablet_bootstrap.cc:654] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.809283 19865 tablet_bootstrap.cc:654] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.809852 19863 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: Bootstrap starting.
I20260809 03:30:10.810456 19866 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: No bootstrap required, opened a new log
I20260809 03:30:10.810542 19866 ts_tablet_manager.cc:1397] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:10.810787 19865 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: No bootstrap required, opened a new log
I20260809 03:30:10.810889 19865 ts_tablet_manager.cc:1397] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 03:30:10.810914 19863 tablet_bootstrap.cc:654] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:10.810998 19866 raft_consensus.cc:348] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.811103 19866 raft_consensus.cc:374] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.811167 19866 raft_consensus.cc:729] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.811297 19866 consensus_queue.cc:260] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.811419 19865 raft_consensus.cc:348] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.811545 19865 raft_consensus.cc:374] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.811560 19866 ts_tablet_manager.cc:1428] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:10.811582 19865 raft_consensus.cc:729] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684aff7e341542edb74eea43e068c195, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.811729 19865 consensus_queue.cc:260] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
W20260809 03:30:10.811832 19595 tablet.cc:2369] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:10.811986 19865 ts_tablet_manager.cc:1428] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:10.812214 19875 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:10.812374 19875 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:10.811543 19876 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:10.812527 19876 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:10.812611 19863 tablet_bootstrap.cc:492] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: No bootstrap required, opened a new log
I20260809 03:30:10.812687 19863 ts_tablet_manager.cc:1397] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:30:10.813048 19863 raft_consensus.cc:348] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.813160 19863 raft_consensus.cc:374] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:10.813211 19863 raft_consensus.cc:729] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Initialized, Role: FOLLOWER
I20260809 03:30:10.813323 19863 consensus_queue.cc:260] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.813536 19874 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:10.813655 19874 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:10.813548 19863 ts_tablet_manager.cc:1428] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260809 03:30:10.843557 19368 tablet.cc:2369] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:10.944859 19876 raft_consensus.cc:482] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:10.945006 19876 raft_consensus.cc:504] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.946233 19876 leader_election.cc:288] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:10.950133 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.946475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:45841 (local address 127.15.222.65:34535)
0809 03:30:10.946770 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.946783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.946906 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.947177 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.947181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.947206 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.947335 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.947342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.948099 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.948102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.949126 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.949136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.949795 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.949798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.949872 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.949897 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.949954 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.950006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
I20260809 03:30:10.950179 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.946555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:34535 (local address 127.15.222.66:45841)
0809 03:30:10.946708 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.946714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.946865 (+ 151us) server_negotiation.cc:407] Connection header received
0809 03:30:10.947018 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.947021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.947075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.947145 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.947432 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.947996 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.949249 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.949654 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.949789 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.949844 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.949979 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.950049 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:30:10.950570 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ce4f74ee94234bc8d049d12440e96" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195" is_pre_election: true
I20260809 03:30:10.950716 19522 raft_consensus.cc:2454] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:10.950788 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.946854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:39305 (local address 127.15.222.65:60043)
0809 03:30:10.947196 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.947208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.947301 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.947526 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.947528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.947545 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:10.947939 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.947945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.948641 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.948644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.949547 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.949556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.950413 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.950417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.950513 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.950535 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:10.950603 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:30:10.950664 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260809 03:30:10.950819 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.946988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:60043 (local address 127.15.222.67:39305)
0809 03:30:10.947135 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.947139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.947282 (+ 143us) server_negotiation.cc:407] Connection header received
0809 03:30:10.947378 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.947380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.947438 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.947512 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:10.948046 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.948538 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.949658 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.950200 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.950309 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.950355 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:10.950635 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 03:30:10.950710 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 03:30:10.951067 19176 leader_election.cc:302] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:10.951383 19876 raft_consensus.cc:2781] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:10.951361 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ce4f74ee94234bc8d049d12440e96" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d" is_pre_election: true
I20260809 03:30:10.951462 19876 raft_consensus.cc:482] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:10.951499 19876 raft_consensus.cc:3037] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.951577 19746 raft_consensus.cc:2454] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:10.952543 19876 raft_consensus.cc:504] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.952821 19876 leader_election.cc:288] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Requested vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:10.953121 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ce4f74ee94234bc8d049d12440e96" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195"
I20260809 03:30:10.953179 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175ce4f74ee94234bc8d049d12440e96" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d"
I20260809 03:30:10.953274 19746 raft_consensus.cc:3037] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.953274 19522 raft_consensus.cc:3037] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:10.954402 19746 raft_consensus.cc:2454] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:10.954407 19522 raft_consensus.cc:2454] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:10.954648 19176 leader_election.cc:302] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723ea1f68a014927a8a4c7fbc894c59d, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:10.954810 19876 raft_consensus.cc:2781] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:10.955010 19876 raft_consensus.cc:686] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [term 1 LEADER]: Becoming Leader. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Running, Role: LEADER
I20260809 03:30:10.955343 19876 consensus_queue.cc:237] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:10.957399 19899 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: TxnStatusTablet state changed. Reason: New leader 7e62ef8993744390890b01c7987d17d6. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:10.957511 19899 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:10.957793 19903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:30:10.957994 19903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:30:10.958417 19031 catalog_manager.cc:5495] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e62ef8993744390890b01c7987d17d6 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:11.000291 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.996651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.0.0.1:48204)
0809 03:30:10.996886 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:10.996900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:10.997012 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:10.997325 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:10.997329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:10.997336 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:10.997451 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.997456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.998172 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.998176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.998797 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:10.998805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.999219 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.999222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:10.999319 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.999361 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.000009 (+ 648us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.000025 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.000087 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.000169 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260809 03:30:11.000291 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:10.996761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.15.222.65:37615)
0809 03:30:10.996899 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:10.996904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:10.996945 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:30:10.997151 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:10.997153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:10.997210 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:10.997293 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:10.997550 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.998069 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.998912 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:10.999118 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:10.999205 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:10.999503 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:10.999913 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.000010 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.000099 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.000150 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:11.005631 19522 raft_consensus.cc:1261] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.005682 19746 raft_consensus.cc:1261] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.006327 19876 consensus_queue.cc:1035] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.006525 19899 consensus_queue.cc:1035] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.009732 19874 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: TxnStatusTablet state changed. Reason: New leader 7e62ef8993744390890b01c7987d17d6. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.009846 19874 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:11.011921 19899 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.012029 19899 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:11.012480 19875 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: TxnStatusTablet state changed. Reason: New leader 7e62ef8993744390890b01c7987d17d6. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.012599 19875 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:11.013809 19874 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.013907 19874 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 723ea1f68a014927a8a4c7fbc894c59d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:11.014657 19906 mvcc.cc:205] Tried to move back new op lower bound from 7316464480271990784 to 7316464480075350016. Current Snapshot: MvccSnapshot[applied={T|T < 7316464480271990784}]
I20260809 03:30:11.016543 19876 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.016644 19876 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 7e62ef8993744390890b01c7987d17d6: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:11.017556 19875 tablet_replica.cc:439] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } } }
I20260809 03:30:11.017653 19875 tablet_replica.cc:442] T 175ce4f74ee94234bc8d049d12440e96 P 684aff7e341542edb74eea43e068c195: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:11.018823 19913 mvcc.cc:205] Tried to move back new op lower bound from 7316464480271990784 to 7316464480075350016. Current Snapshot: MvccSnapshot[applied={T|T < 7316464480271990784}]
I20260809 03:30:11.023726 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.020056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.15.222.67:39305)
0809 03:30:11.020159 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.020162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.020174 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:11.020317 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.020320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.020376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.020485 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.020678 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.021436 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.022274 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.022630 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.022732 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.022928 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.023381 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.023496 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.023546 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.023600 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 03:30:11.033169 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.030305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.15.222.67:57835)
0809 03:30:11.030434 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.030448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.030527 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.030730 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.030733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.030740 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:11.030854 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.030861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.031527 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.031530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.032106 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.032113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.032516 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.032519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.032607 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.032647 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.032923 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.032929 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.032987 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.033063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 03:30:11.033164 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.030375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:57835 (local address 127.15.222.65:37615)
0809 03:30:11.030484 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.030488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.030500 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:11.030616 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.030619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.030658 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.030734 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.030964 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.031433 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.032212 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.032425 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.032506 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.032786 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.032848 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.032905 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.032995 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.033042 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:30:11.039245 19522 raft_consensus.cc:1261] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Refusing update from remote peer 723ea1f68a014927a8a4c7fbc894c59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.039297 19294 raft_consensus.cc:1261] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Refusing update from remote peer 723ea1f68a014927a8a4c7fbc894c59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.039667 19880 consensus_queue.cc:1035] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.039834 19874 consensus_queue.cc:1035] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.042721 19909 mvcc.cc:205] Tried to move back new op lower bound from 7316464480412684288 to 7316464479248044032. Current Snapshot: MvccSnapshot[applied={T|T < 7316464480412684288}]
I20260809 03:30:11.062078 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.058180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:60425 (local address 127.15.222.67:39305)
0809 03:30:11.058291 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.058295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.058595 (+ 300us) server_negotiation.cc:407] Connection header received
0809 03:30:11.058781 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.058785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.058833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.058876 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.059258 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.059750 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.060853 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.061130 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.061243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.061644 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.061735 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.061798 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.061877 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.061957 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:30:11.062101 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.058086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:39305 (local address 127.15.222.65:60425)
0809 03:30:11.058512 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.058527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.058669 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.059033 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.059036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.059044 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:11.059140 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.059146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.059837 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.059841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.060742 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.060754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.061347 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.061352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.061445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.061509 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.061803 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.061809 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.061868 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.061948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260809 03:30:11.092555 19030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:11.092762 19030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:11.095664 19030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Sending DeleteTablet for 3 replicas of tablet 8b04c27b9b9743748120deba81a8ff05
I20260809 03:30:11.096985 19494 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b04c27b9b9743748120deba81a8ff05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:11 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42638
I20260809 03:30:11.097469 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b04c27b9b9743748120deba81a8ff05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:11 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47504
I20260809 03:30:11.098610 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b04c27b9b9743748120deba81a8ff05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:11 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:48192
I20260809 03:30:11.099079 19030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:11.099509 19030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:11.101883 19950 tablet_replica.cc:330] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: stopping tablet replica
I20260809 03:30:11.102178 19950 raft_consensus.cc:2229] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:11.102516 19950 raft_consensus.cc:2258] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:11.104512 19715 tablet_service.cc:1467] Processing CreateTablet for tablet b152137c726441da8f70381a664d212a (DEFAULT_TABLE table=test-table [id=6b3dbeeac27248859faed311b05303d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:11.104794 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b152137c726441da8f70381a664d212a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:11.107327 19494 tablet_service.cc:1467] Processing CreateTablet for tablet b152137c726441da8f70381a664d212a (DEFAULT_TABLE table=test-table [id=6b3dbeeac27248859faed311b05303d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:11.107609 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b152137c726441da8f70381a664d212a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:11.109846 19950 ts_tablet_manager.cc:1905] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:11.110167 19263 tablet_service.cc:1467] Processing CreateTablet for tablet b152137c726441da8f70381a664d212a (DEFAULT_TABLE table=test-table [id=6b3dbeeac27248859faed311b05303d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:11.110443 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b152137c726441da8f70381a664d212a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:11.111996 19952 tablet_replica.cc:330] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: stopping tablet replica
I20260809 03:30:11.112298 19952 raft_consensus.cc:2229] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:11.112469 19952 raft_consensus.cc:2258] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:11.112843 19866 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Bootstrap starting.
I20260809 03:30:11.113674 19866 tablet_bootstrap.cc:654] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:11.113785 19952 ts_tablet_manager.cc:1905] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:11.114833 19865 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Bootstrap starting.
I20260809 03:30:11.115746 19865 tablet_bootstrap.cc:654] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:11.116487 19952 ts_tablet_manager.cc:1918] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:11.116580 19952 log.cc:1196] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/8b04c27b9b9743748120deba81a8ff05
I20260809 03:30:11.116735 19863 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Bootstrap starting.
I20260809 03:30:11.116849 19952 ts_tablet_manager.cc:1939] T 8b04c27b9b9743748120deba81a8ff05 P 7e62ef8993744390890b01c7987d17d6: Deleting consensus metadata
I20260809 03:30:11.117656 19863 tablet_bootstrap.cc:654] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:11.118801 19950 ts_tablet_manager.cc:1918] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:11.118899 19950 log.cc:1196] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/8b04c27b9b9743748120deba81a8ff05
I20260809 03:30:11.119182 19950 ts_tablet_manager.cc:1939] T 8b04c27b9b9743748120deba81a8ff05 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting consensus metadata
I20260809 03:30:11.120213 19010 catalog_manager.cc:4841] TS 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305): tablet 8b04c27b9b9743748120deba81a8ff05 (table test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]) successfully deleted
I20260809 03:30:11.120430 19866 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: No bootstrap required, opened a new log
I20260809 03:30:11.120517 19866 ts_tablet_manager.cc:1397] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260809 03:30:11.120905 19866 raft_consensus.cc:348] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.121024 19866 raft_consensus.cc:374] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:11.121079 19866 raft_consensus.cc:729] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Initialized, Role: FOLLOWER
I20260809 03:30:11.121194 19866 consensus_queue.cc:260] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.121448 19866 ts_tablet_manager.cc:1428] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:11.121904 19007 catalog_manager.cc:4841] TS 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615): tablet 8b04c27b9b9743748120deba81a8ff05 (table test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]) successfully deleted
I20260809 03:30:11.122341 19949 tablet_replica.cc:330] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: stopping tablet replica
I20260809 03:30:11.122685 19949 raft_consensus.cc:2229] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:11.122781 19863 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: No bootstrap required, opened a new log
I20260809 03:30:11.122867 19863 ts_tablet_manager.cc:1397] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 03:30:11.122990 19949 raft_consensus.cc:2258] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:11.123270 19863 raft_consensus.cc:348] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.123374 19863 raft_consensus.cc:374] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:11.123416 19863 raft_consensus.cc:729] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Initialized, Role: FOLLOWER
I20260809 03:30:11.123907 19865 tablet_bootstrap.cc:492] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: No bootstrap required, opened a new log
I20260809 03:30:11.123994 19865 ts_tablet_manager.cc:1397] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260809 03:30:11.124416 19865 raft_consensus.cc:348] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.124527 19865 raft_consensus.cc:374] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:11.124562 19865 raft_consensus.cc:729] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684aff7e341542edb74eea43e068c195, State: Initialized, Role: FOLLOWER
I20260809 03:30:11.124677 19865 consensus_queue.cc:260] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.124745 19949 ts_tablet_manager.cc:1905] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:11.124917 19865 ts_tablet_manager.cc:1428] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:11.127542 19949 ts_tablet_manager.cc:1918] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:11.127686 19949 log.cc:1196] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/8b04c27b9b9743748120deba81a8ff05
I20260809 03:30:11.127949 19863 consensus_queue.cc:260] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.128024 19949 ts_tablet_manager.cc:1939] T 8b04c27b9b9743748120deba81a8ff05 P 684aff7e341542edb74eea43e068c195: Deleting consensus metadata
I20260809 03:30:11.128242 19863 ts_tablet_manager.cc:1428] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260809 03:30:11.129634 19010 catalog_manager.cc:4841] TS 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841): tablet 8b04c27b9b9743748120deba81a8ff05 (table test-table [id=eb8c80cab1a2408588b46ef060b5c6bd]) successfully deleted
I20260809 03:30:11.332820 19875 raft_consensus.cc:482] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:11.332957 19875 raft_consensus.cc:504] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.333947 19875 leader_election.cc:288] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:11.337569 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.334268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:56037 (local address 127.15.222.67:39305)
0809 03:30:11.334394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.334399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.334613 (+ 214us) server_negotiation.cc:407] Connection header received
0809 03:30:11.334733 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.334736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.334780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.334840 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:11.335137 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.335612 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.336625 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.337000 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.337102 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.337160 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.337371 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.337447 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:11.337568 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.334255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:36355 (local address 127.15.222.65:37615)
0809 03:30:11.334378 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.334382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.334583 (+ 201us) server_negotiation.cc:407] Connection header received
0809 03:30:11.334698 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.334701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.334745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.334805 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:11.335089 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.335585 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.336633 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.337020 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.337123 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.337175 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.337375 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.337442 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:11.337598 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.334161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:39305 (local address 127.15.222.66:56037)
0809 03:30:11.334522 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.334533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.334636 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.334856 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.334859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.334905 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:11.335026 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.335032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.335725 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.335729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.336513 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.336523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.337111 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.337115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.337251 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.337281 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.337343 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.337425 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":134,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260809 03:30:11.337613 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.334161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.15.222.66:36355)
0809 03:30:11.334479 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.334492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.334607 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.334821 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.334823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.334839 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:11.334982 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.334987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.335683 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.335686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.336501 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.336511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.337123 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.337127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.337248 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.337285 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.337346 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.337443 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":60,"spinlock_wait_cycles":28928,"thread_start_us":66,"threads_started":1}
I20260809 03:30:11.338059 19294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b152137c726441da8f70381a664d212a" candidate_uuid: "684aff7e341542edb74eea43e068c195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e62ef8993744390890b01c7987d17d6" is_pre_election: true
I20260809 03:30:11.338201 19294 raft_consensus.cc:2454] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 684aff7e341542edb74eea43e068c195 in term 0.
I20260809 03:30:11.338359 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b152137c726441da8f70381a664d212a" candidate_uuid: "684aff7e341542edb74eea43e068c195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d" is_pre_election: true
I20260809 03:30:11.338524 19746 raft_consensus.cc:2454] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 684aff7e341542edb74eea43e068c195 in term 0.
I20260809 03:30:11.338855 19411 leader_election.cc:302] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:11.339107 19875 raft_consensus.cc:2781] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:11.339166 19875 raft_consensus.cc:482] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:11.339193 19875 raft_consensus.cc:3037] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:11.340250 19875 raft_consensus.cc:504] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.340580 19875 leader_election.cc:288] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [CANDIDATE]: Term 1 election: Requested vote from peers 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:11.340867 19294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b152137c726441da8f70381a664d212a" candidate_uuid: "684aff7e341542edb74eea43e068c195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e62ef8993744390890b01c7987d17d6"
I20260809 03:30:11.340909 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b152137c726441da8f70381a664d212a" candidate_uuid: "684aff7e341542edb74eea43e068c195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d"
I20260809 03:30:11.340973 19294 raft_consensus.cc:3037] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:11.341024 19746 raft_consensus.cc:3037] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:11.341881 19294 raft_consensus.cc:2454] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 684aff7e341542edb74eea43e068c195 in term 1.
I20260809 03:30:11.341985 19746 raft_consensus.cc:2454] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 684aff7e341542edb74eea43e068c195 in term 1.
I20260809 03:30:11.342133 19411 leader_election.cc:302] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:11.342338 19875 raft_consensus.cc:2781] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:11.342520 19875 raft_consensus.cc:686] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 1 LEADER]: Becoming Leader. State: Replica: 684aff7e341542edb74eea43e068c195, State: Running, Role: LEADER
I20260809 03:30:11.342756 19875 consensus_queue.cc:237] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:11.344777 19030 catalog_manager.cc:5495] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 reported cstate change: term changed from 0 to 1, leader changed from <none> to 684aff7e341542edb74eea43e068c195 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "684aff7e341542edb74eea43e068c195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:11.370011 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.366886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.15.222.66:45841)
0809 03:30:11.366997 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.367000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.367011 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:30:11.367131 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.367134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.367191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.367276 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.367480 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.367991 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.368729 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.368947 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.369018 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.369213 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.369691 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.369787 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.369848 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.369902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:11.379155 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.376485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:58675 (local address 127.15.222.65:37615)
0809 03:30:11.376588 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.376591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.376727 (+ 136us) server_negotiation.cc:407] Connection header received
0809 03:30:11.376827 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.376830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.376876 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.376935 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.377121 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.377597 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.378355 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.378568 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.378635 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.378845 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.378904 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.378949 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.379011 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.379052 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:30:11.379177 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.376376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.15.222.66:58675)
0809 03:30:11.376658 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.376670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.376754 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.376949 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.376951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.376957 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:11.377049 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.377053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.377669 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.377671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.378274 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.378282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.378645 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.378647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.378710 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.378750 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.378961 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.378965 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.379004 (+ 39us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.379061 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260809 03:30:11.384315 19746 raft_consensus.cc:1261] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Refusing update from remote peer 684aff7e341542edb74eea43e068c195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.384573 19294 raft_consensus.cc:1261] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Refusing update from remote peer 684aff7e341542edb74eea43e068c195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:11.384727 19875 consensus_queue.cc:1035] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.384941 19966 consensus_queue.cc:1035] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:11.400571 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.397594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:36025 (local address 127.15.222.66:45841)
0809 03:30:11.397691 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:11.397695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:11.397746 (+ 51us) server_negotiation.cc:407] Connection header received
0809 03:30:11.397898 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:11.397901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:11.397944 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:11.398000 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:11.398228 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.398719 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.399485 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.399762 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.399857 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.400152 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.400227 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.400294 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:11.400399 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 03:30:11.400447 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:30:11.400589 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:11.397525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:45841 (local address 127.15.222.65:36025)
0809 03:30:11.397674 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:11.397688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:11.397804 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:11.398022 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:11.398025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:11.398032 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:11.398122 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.398128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.398804 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.398806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.399393 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:11.399403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:11.399850 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:11.399853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:11.399925 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:11.399951 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:11.400308 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:11.400314 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:11.400386 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:30:11.400469 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
W20260809 03:30:11.401177 19913 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 03:30:12.956288 19031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:12.956441 19031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:12.958665 19031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Sending DeleteTablet for 3 replicas of tablet b152137c726441da8f70381a664d212a
I20260809 03:30:12.959139 19261 tablet_service.cc:1514] Processing DeleteTablet for tablet b152137c726441da8f70381a664d212a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:12 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:48192
I20260809 03:30:12.959336 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet b152137c726441da8f70381a664d212a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:12 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47504
I20260809 03:30:12.959699 20014 tablet_replica.cc:330] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: stopping tablet replica
I20260809 03:30:12.959877 20014 raft_consensus.cc:2229] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:12.960041 20014 raft_consensus.cc:2258] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:12.959450 20012 tablet_replica.cc:330] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: stopping tablet replica
I20260809 03:30:12.960187 20012 raft_consensus.cc:2229] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:12.960294 20012 raft_consensus.cc:2258] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:12.960801 20014 ts_tablet_manager.cc:1905] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:12.960814 20012 ts_tablet_manager.cc:1905] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:12.962574 19494 tablet_service.cc:1514] Processing DeleteTablet for tablet b152137c726441da8f70381a664d212a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:12 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42638
I20260809 03:30:12.962898 20017 tablet_replica.cc:330] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: stopping tablet replica
I20260809 03:30:12.963022 20017 raft_consensus.cc:2229] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:12.963322 20017 raft_consensus.cc:2258] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:12.963812 20012 ts_tablet_manager.cc:1918] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:30:12.963881 20012 log.cc:1196] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/b152137c726441da8f70381a664d212a
I20260809 03:30:12.963871 20017 ts_tablet_manager.cc:1905] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:12.964120 20012 ts_tablet_manager.cc:1939] T b152137c726441da8f70381a664d212a P 7e62ef8993744390890b01c7987d17d6: Deleting consensus metadata
I20260809 03:30:12.964509 20014 ts_tablet_manager.cc:1918] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:30:12.964576 20014 log.cc:1196] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/b152137c726441da8f70381a664d212a
I20260809 03:30:12.964689 19007 catalog_manager.cc:4841] TS 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615): tablet b152137c726441da8f70381a664d212a (table test-table [id=6b3dbeeac27248859faed311b05303d0]) successfully deleted
I20260809 03:30:12.964794 20014 ts_tablet_manager.cc:1939] T b152137c726441da8f70381a664d212a P 723ea1f68a014927a8a4c7fbc894c59d: Deleting consensus metadata
I20260809 03:30:12.965411 19010 catalog_manager.cc:4841] TS 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305): tablet b152137c726441da8f70381a664d212a (table test-table [id=6b3dbeeac27248859faed311b05303d0]) successfully deleted
I20260809 03:30:12.966320 20017 ts_tablet_manager.cc:1918] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:30:12.966393 20017 log.cc:1196] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/b152137c726441da8f70381a664d212a
I20260809 03:30:12.966639 20017 ts_tablet_manager.cc:1939] T b152137c726441da8f70381a664d212a P 684aff7e341542edb74eea43e068c195: Deleting consensus metadata
I20260809 03:30:12.967255 19010 catalog_manager.cc:4841] TS 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841): tablet b152137c726441da8f70381a664d212a (table test-table [id=6b3dbeeac27248859faed311b05303d0]) successfully deleted
Aug 09 03:30:12 dist-test-slave-cvwc krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:30:12 dist-test-slave-cvwc krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:12.987679 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:12.980221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.15.222.126:34335)
0809 03:30:12.980474 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:12.980481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:12.980494 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:12.980604 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:12.980606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:12.980648 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:12.980724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:12.981314 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.981947 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.982723 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:12.982936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:12.982999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:12.985151 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:12.985154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:12.985156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:12.985179 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:12.986409 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:12.986747 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:12.986750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:12.986751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:12.986819 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:12.987005 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:12.987007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:12.987008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:12.987162 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:12.987255 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:12.987421 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 03:30:12.987515 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
I20260809 03:30:12.990031 19031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:12.990448 19031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:12.994808 19261 tablet_service.cc:1467] Processing CreateTablet for tablet 89046aa599284bdc97012ff27394e37d (DEFAULT_TABLE table=test-table [id=5c9ff673ea384ae4a660253a05dc985d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:12.994951 19494 tablet_service.cc:1467] Processing CreateTablet for tablet 89046aa599284bdc97012ff27394e37d (DEFAULT_TABLE table=test-table [id=5c9ff673ea384ae4a660253a05dc985d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:12.995085 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 89046aa599284bdc97012ff27394e37d (DEFAULT_TABLE table=test-table [id=5c9ff673ea384ae4a660253a05dc985d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:12.995152 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89046aa599284bdc97012ff27394e37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:12.995234 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89046aa599284bdc97012ff27394e37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:12.995330 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89046aa599284bdc97012ff27394e37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:12.997624 20034 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Bootstrap starting.
I20260809 03:30:12.997936 20038 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Bootstrap starting.
I20260809 03:30:12.997936 20032 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Bootstrap starting.
I20260809 03:30:12.998490 20034 tablet_bootstrap.cc:654] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:12.998765 20038 tablet_bootstrap.cc:654] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:12.998764 20032 tablet_bootstrap.cc:654] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:12.999571 20034 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: No bootstrap required, opened a new log
I20260809 03:30:12.999656 20034 ts_tablet_manager.cc:1397] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:12.999814 20032 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: No bootstrap required, opened a new log
I20260809 03:30:12.999897 20032 ts_tablet_manager.cc:1397] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:13.000038 20034 raft_consensus.cc:348] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.000156 20034 raft_consensus.cc:374] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.000191 20034 raft_consensus.cc:729] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684aff7e341542edb74eea43e068c195, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.000248 20032 raft_consensus.cc:348] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.000342 20032 raft_consensus.cc:374] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.000372 20032 raft_consensus.cc:729] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.000319 20034 consensus_queue.cc:260] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.000478 20032 consensus_queue.cc:260] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.000676 20034 ts_tablet_manager.cc:1428] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:13.000727 20032 ts_tablet_manager.cc:1428] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:13.001158 20038 tablet_bootstrap.cc:492] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: No bootstrap required, opened a new log
I20260809 03:30:13.001241 20038 ts_tablet_manager.cc:1397] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 03:30:13.001624 20038 raft_consensus.cc:348] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.001765 20038 raft_consensus.cc:374] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.001804 20038 raft_consensus.cc:729] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.001930 20038 consensus_queue.cc:260] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.002296 20038 ts_tablet_manager.cc:1428] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:13.014469 19985 raft_consensus.cc:482] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:13.014571 19985 raft_consensus.cc:504] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.014917 19985 leader_election.cc:288] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:13.015244 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89046aa599284bdc97012ff27394e37d" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195" is_pre_election: true
I20260809 03:30:13.015291 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89046aa599284bdc97012ff27394e37d" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d" is_pre_election: true
I20260809 03:30:13.015370 19522 raft_consensus.cc:2454] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:13.015420 19746 raft_consensus.cc:2454] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:13.015643 19176 leader_election.cc:302] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:13.015807 19985 raft_consensus.cc:2781] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:13.015864 19985 raft_consensus.cc:482] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:13.015892 19985 raft_consensus.cc:3037] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.016813 19985 raft_consensus.cc:504] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.017093 19985 leader_election.cc:288] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Requested vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:13.017391 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89046aa599284bdc97012ff27394e37d" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195"
I20260809 03:30:13.017429 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89046aa599284bdc97012ff27394e37d" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d"
I20260809 03:30:13.017505 19522 raft_consensus.cc:3037] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.017535 19746 raft_consensus.cc:3037] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.018604 19746 raft_consensus.cc:2454] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:13.018604 19522 raft_consensus.cc:2454] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:13.018905 19176 leader_election.cc:302] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723ea1f68a014927a8a4c7fbc894c59d, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:13.019145 19985 raft_consensus.cc:2781] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:13.019225 19985 raft_consensus.cc:686] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 1 LEADER]: Becoming Leader. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Running, Role: LEADER
I20260809 03:30:13.019356 19985 consensus_queue.cc:237] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.020756 19031 catalog_manager.cc:5495] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e62ef8993744390890b01c7987d17d6 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:13.036814 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.034120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.15.222.65:37615)
0809 03:30:13.034368 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:13.034375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:13.034388 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:13.034425 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:13.034427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:13.034465 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:13.034533 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:13.034723 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.035244 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.036005 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.036227 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.036295 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.036464 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:13.036527 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:13.036571 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:13.036628 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 03:30:13.036668 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260809 03:30:13.044505 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.042061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:37075 (local address 127.15.222.65:37615)
0809 03:30:13.042160 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:13.042163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:13.042203 (+ 40us) server_negotiation.cc:407] Connection header received
0809 03:30:13.042306 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:13.042308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:13.042346 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:13.042404 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:13.042589 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.043045 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.043747 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.043946 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.044031 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.044191 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:13.044246 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:13.044293 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:13.044350 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 03:30:13.044388 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":41}
I20260809 03:30:13.044519 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:13.041907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.15.222.65:37075)
0809 03:30:13.042140 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:13.042153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:13.042238 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:13.042417 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:13.042419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:13.042425 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:13.042522 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.042526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.043112 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.043114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.043668 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:13.043674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:13.044019 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:13.044022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:13.044092 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:13.044114 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:13.044299 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:13.044302 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:13.044343 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 03:30:13.044401 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260809 03:30:13.049197 19746 raft_consensus.cc:1261] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:13.049232 19522 raft_consensus.cc:1261] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:13.049569 19993 consensus_queue.cc:1035] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:13.049706 19985 consensus_queue.cc:1035] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:13.080873 19033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:13.081053 19033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:13.083137 19033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Sending DeleteTablet for 3 replicas of tablet 89046aa599284bdc97012ff27394e37d
I20260809 03:30:13.083621 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 89046aa599284bdc97012ff27394e37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:13 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:48192
I20260809 03:30:13.083753 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 89046aa599284bdc97012ff27394e37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:13 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47504
I20260809 03:30:13.083963 20012 tablet_replica.cc:330] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: stopping tablet replica
I20260809 03:30:13.084089 20012 raft_consensus.cc:2229] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:13.084374 20012 raft_consensus.cc:2258] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:13.084439 19033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:30:13.084578 20014 tablet_replica.cc:330] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: stopping tablet replica
I20260809 03:30:13.084685 20014 raft_consensus.cc:2229] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:13.084796 20014 raft_consensus.cc:2258] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 03:30:13.084831 19033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:13.085254 19494 tablet_service.cc:1514] Processing DeleteTablet for tablet 89046aa599284bdc97012ff27394e37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:13 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42638
I20260809 03:30:13.085438 20017 tablet_replica.cc:330] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: stopping tablet replica
I20260809 03:30:13.085537 20017 raft_consensus.cc:2229] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:13.085636 20017 raft_consensus.cc:2258] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:13.085801 20012 ts_tablet_manager.cc:1905] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:13.086256 20014 ts_tablet_manager.cc:1905] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:13.088344 20012 ts_tablet_manager.cc:1918] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:13.088430 20012 log.cc:1196] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/89046aa599284bdc97012ff27394e37d
I20260809 03:30:13.088688 20012 ts_tablet_manager.cc:1939] T 89046aa599284bdc97012ff27394e37d P 7e62ef8993744390890b01c7987d17d6: Deleting consensus metadata
I20260809 03:30:13.088781 20014 ts_tablet_manager.cc:1918] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:13.088841 20014 log.cc:1196] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/89046aa599284bdc97012ff27394e37d
I20260809 03:30:13.089056 20014 ts_tablet_manager.cc:1939] T 89046aa599284bdc97012ff27394e37d P 723ea1f68a014927a8a4c7fbc894c59d: Deleting consensus metadata
I20260809 03:30:13.089342 19007 catalog_manager.cc:4841] TS 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615): tablet 89046aa599284bdc97012ff27394e37d (table test-table [id=5c9ff673ea384ae4a660253a05dc985d]) successfully deleted
I20260809 03:30:13.090173 20017 ts_tablet_manager.cc:1905] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:13.092005 20017 ts_tablet_manager.cc:1918] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:13.092072 20017 log.cc:1196] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/89046aa599284bdc97012ff27394e37d
I20260809 03:30:13.092238 20017 ts_tablet_manager.cc:1939] T 89046aa599284bdc97012ff27394e37d P 684aff7e341542edb74eea43e068c195: Deleting consensus metadata
I20260809 03:30:13.093101 19010 catalog_manager.cc:4841] TS 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841): tablet 89046aa599284bdc97012ff27394e37d (table test-table [id=5c9ff673ea384ae4a660253a05dc985d]) successfully deleted
I20260809 03:30:13.094766 19010 catalog_manager.cc:4841] TS 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305): tablet 89046aa599284bdc97012ff27394e37d (table test-table [id=5c9ff673ea384ae4a660253a05dc985d]) successfully deleted
I20260809 03:30:13.096863 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 268673233f3349ebb99b308b7bb9bda7 (DEFAULT_TABLE table=test-table [id=834bd9c25e2943aea48e2709c5d2a346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:13.097121 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268673233f3349ebb99b308b7bb9bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:13.097121 19494 tablet_service.cc:1467] Processing CreateTablet for tablet 268673233f3349ebb99b308b7bb9bda7 (DEFAULT_TABLE table=test-table [id=834bd9c25e2943aea48e2709c5d2a346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:13.097366 19494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268673233f3349ebb99b308b7bb9bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:13.099349 20032 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6: Bootstrap starting.
I20260809 03:30:13.099507 20034 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Bootstrap starting.
I20260809 03:30:13.100243 20032 tablet_bootstrap.cc:654] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:13.100298 20034 tablet_bootstrap.cc:654] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:13.101119 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 268673233f3349ebb99b308b7bb9bda7 (DEFAULT_TABLE table=test-table [id=834bd9c25e2943aea48e2709c5d2a346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:13.101372 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268673233f3349ebb99b308b7bb9bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:13.102850 20032 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6: No bootstrap required, opened a new log
I20260809 03:30:13.102948 20032 ts_tablet_manager.cc:1397] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:30:13.103327 20032 raft_consensus.cc:348] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.103439 20032 raft_consensus.cc:374] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.103475 20032 raft_consensus.cc:729] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.103595 20032 consensus_queue.cc:260] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.103636 20038 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Bootstrap starting.
I20260809 03:30:13.103842 20032 ts_tablet_manager.cc:1428] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:13.104486 20038 tablet_bootstrap.cc:654] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:13.105527 20034 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: No bootstrap required, opened a new log
I20260809 03:30:13.105611 20034 ts_tablet_manager.cc:1397] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260809 03:30:13.106068 20034 raft_consensus.cc:348] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.106178 20034 raft_consensus.cc:374] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.106210 20034 raft_consensus.cc:729] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684aff7e341542edb74eea43e068c195, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.106317 20034 consensus_queue.cc:260] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.106554 20034 ts_tablet_manager.cc:1428] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:13.107650 20038 tablet_bootstrap.cc:492] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: No bootstrap required, opened a new log
I20260809 03:30:13.107731 20038 ts_tablet_manager.cc:1397] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 03:30:13.108096 20038 raft_consensus.cc:348] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.108198 20038 raft_consensus.cc:374] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:13.108233 20038 raft_consensus.cc:729] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ea1f68a014927a8a4c7fbc894c59d, State: Initialized, Role: FOLLOWER
I20260809 03:30:13.108343 20038 consensus_queue.cc:260] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.108592 20038 ts_tablet_manager.cc:1428] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:13.393831 19930 raft_consensus.cc:482] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:13.393997 19930 raft_consensus.cc:504] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.394378 19930 leader_election.cc:288] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:13.394840 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268673233f3349ebb99b308b7bb9bda7" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195" is_pre_election: true
I20260809 03:30:13.394882 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268673233f3349ebb99b308b7bb9bda7" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d" is_pre_election: true
I20260809 03:30:13.395004 19522 raft_consensus.cc:2454] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:13.395016 19746 raft_consensus.cc:2454] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 0.
I20260809 03:30:13.395318 19176 leader_election.cc:302] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:13.395520 19930 raft_consensus.cc:2781] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:13.395576 19930 raft_consensus.cc:482] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:13.395601 19930 raft_consensus.cc:3037] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.396606 19930 raft_consensus.cc:504] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.396914 19930 leader_election.cc:288] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Requested vote from peers 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841), 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305)
I20260809 03:30:13.397235 19522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268673233f3349ebb99b308b7bb9bda7" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684aff7e341542edb74eea43e068c195"
I20260809 03:30:13.397297 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268673233f3349ebb99b308b7bb9bda7" candidate_uuid: "7e62ef8993744390890b01c7987d17d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723ea1f68a014927a8a4c7fbc894c59d"
I20260809 03:30:13.397352 19522 raft_consensus.cc:3037] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.397426 19746 raft_consensus.cc:3037] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:13.398317 19522 raft_consensus.cc:2454] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:13.398356 19746 raft_consensus.cc:2454] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e62ef8993744390890b01c7987d17d6 in term 1.
I20260809 03:30:13.398584 19176 leader_election.cc:302] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684aff7e341542edb74eea43e068c195, 7e62ef8993744390890b01c7987d17d6; no voters:
I20260809 03:30:13.398756 19930 raft_consensus.cc:2781] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:13.398844 19930 raft_consensus.cc:686] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [term 1 LEADER]: Becoming Leader. State: Replica: 7e62ef8993744390890b01c7987d17d6, State: Running, Role: LEADER
I20260809 03:30:13.398983 19930 consensus_queue.cc:237] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } }
I20260809 03:30:13.400564 19030 catalog_manager.cc:5495] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e62ef8993744390890b01c7987d17d6 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "7e62ef8993744390890b01c7987d17d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e62ef8993744390890b01c7987d17d6" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:13.468504 19746 raft_consensus.cc:1261] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:13.468474 19522 raft_consensus.cc:1261] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Refusing update from remote peer 7e62ef8993744390890b01c7987d17d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:13.468839 19930 consensus_queue.cc:1035] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723ea1f68a014927a8a4c7fbc894c59d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:13.469002 19993 consensus_queue.cc:1035] T 268673233f3349ebb99b308b7bb9bda7 P 7e62ef8993744390890b01c7987d17d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684aff7e341542edb74eea43e068c195" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:30:13.481524 19983 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 03:30:15.028317 19030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:15.028479 19030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:15.030750 19030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2def972f29ad44a4ae66fec0b1062ec2: Sending DeleteTablet for 3 replicas of tablet 268673233f3349ebb99b308b7bb9bda7
I20260809 03:30:15.031268 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 268673233f3349ebb99b308b7bb9bda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:48192
I20260809 03:30:15.031268 19494 tablet_service.cc:1514] Processing DeleteTablet for tablet 268673233f3349ebb99b308b7bb9bda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42638
I20260809 03:30:15.031358 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 268673233f3349ebb99b308b7bb9bda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:15 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47504
I20260809 03:30:15.031651 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19150
I20260809 03:30:15.038996 20092 tablet_replica.cc:330] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: stopping tablet replica
I20260809 03:30:15.039147 20092 raft_consensus.cc:2229] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:15.039268 20092 raft_consensus.cc:2258] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:15.039947 20092 ts_tablet_manager.cc:1905] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:15.041998 20094 tablet_replica.cc:330] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: stopping tablet replica
I20260809 03:30:15.042131 20094 raft_consensus.cc:2229] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:15.042150 20092 ts_tablet_manager.cc:1918] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:30:15.042220 20092 log.cc:1196] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/268673233f3349ebb99b308b7bb9bda7
I20260809 03:30:15.042243 20094 raft_consensus.cc:2258] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:15.042501 20092 ts_tablet_manager.cc:1939] T 268673233f3349ebb99b308b7bb9bda7 P 684aff7e341542edb74eea43e068c195: Deleting consensus metadata
I20260809 03:30:15.042821 20094 ts_tablet_manager.cc:1905] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:15.043150 19010 catalog_manager.cc:4841] TS 684aff7e341542edb74eea43e068c195 (127.15.222.66:45841): tablet 268673233f3349ebb99b308b7bb9bda7 (table test-table [id=834bd9c25e2943aea48e2709c5d2a346]) successfully deleted
I20260809 03:30:15.045329 20094 ts_tablet_manager.cc:1918] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 03:30:15.045423 20094 log.cc:1196] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/268673233f3349ebb99b308b7bb9bda7
I20260809 03:30:15.045696 20094 ts_tablet_manager.cc:1939] T 268673233f3349ebb99b308b7bb9bda7 P 723ea1f68a014927a8a4c7fbc894c59d: Deleting consensus metadata
I20260809 03:30:15.046363 19010 catalog_manager.cc:4841] TS 723ea1f68a014927a8a4c7fbc894c59d (127.15.222.67:39305): tablet 268673233f3349ebb99b308b7bb9bda7 (table test-table [id=834bd9c25e2943aea48e2709c5d2a346]) successfully deleted
W20260809 03:30:15.048286 19007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:37615 (error 108)
I20260809 03:30:15.048503 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19385
I20260809 03:30:15.055203 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:15.053756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37615 (local address 127.0.0.1:48798)
0809 03:30:15.055047 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:15.055120 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":1130,"threads_started":1}
W20260809 03:30:15.055330 19007 catalog_manager.cc:4584] TS 7e62ef8993744390890b01c7987d17d6 (127.15.222.65:37615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 268673233f3349ebb99b308b7bb9bda7: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37615: connect: Connection refused (error 111)
I20260809 03:30:15.060879 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19609
I20260809 03:30:15.071970 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18978
2026-08-09T03:30:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6737 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:15 dist-test-slave-cvwc krb5kdc[20102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:15 dist-test-slave-cvwc krb5kdc[20102](info): set up 2 sockets
Aug 09 03:30:15 dist-test-slave-cvwc krb5kdc[20102](info): commencing operation
krb5kdc: starting...
W20260809 03:30:16.216858 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:16.328550 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46253
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:16.435318 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:16.435565 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:16.435621 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:16.439589 20122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:16.439646 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:16.439672 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:16.439693 20122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:16.439723 20122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:16.444731 20122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:16.445672 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:16.446643 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:16.454733 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:16.454771 20122 server_base.cc:1034] running on GCE node
W20260809 03:30:16.454723 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:16.454723 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:16.455402 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:16.456362 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:16.457517 20122 hybrid_clock.cc:648] HybridClock initialized: now 1786246216457495 us; error 34 us; skew 500 ppm
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:16.460091 20122 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:16.460893 20122 webserver.cc:460] Webserver started at http://127.15.222.126:35211/ using document root <none> and password file <none>
I20260809 03:30:16.461416 20122 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:16.461477 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:16.461674 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:16.463624 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "4793cdef8b514193ace87997a9ac7755"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "fa8968b590b2023b191d01f562e0b919"
server_key_iv: "16634bfdb7bb5250db8755e591e4c175"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.464151 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "4793cdef8b514193ace87997a9ac7755"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "fa8968b590b2023b191d01f562e0b919"
server_key_iv: "16634bfdb7bb5250db8755e591e4c175"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.467842 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:16.470134 20151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.471251 20122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:16.471354 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "4793cdef8b514193ace87997a9ac7755"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "fa8968b590b2023b191d01f562e0b919"
server_key_iv: "16634bfdb7bb5250db8755e591e4c175"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.471457 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:16.494890 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:16.497525 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:16.497699 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:16.506076 20122 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46253
I20260809 03:30:16.506103 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46253 every 8 connection(s)
I20260809 03:30:16.507014 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:16.509847 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:16.514900 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 20122
I20260809 03:30:16.515010 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:16.515277 16249 external_mini_cluster.cc:1428] Setting key d0a3429fba98281133372bdf48ca9333
I20260809 03:30:16.515398 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: Bootstrap starting.
I20260809 03:30:16.517915 20234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:16.518625 20234 log.cc:824] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:16.520308 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: No bootstrap required, opened a new log
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:16.523734 20234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } }
I20260809 03:30:16.523948 20234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:16.524005 20234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4793cdef8b514193ace87997a9ac7755, State: Initialized, Role: FOLLOWER
I20260809 03:30:16.524448 20234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } }
I20260809 03:30:16.524591 20234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:16.524655 20234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:16.524747 20234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:16.525496 20234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } }
I20260809 03:30:16.525840 20234 leader_election.cc:302] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4793cdef8b514193ace87997a9ac7755; no voters:
I20260809 03:30:16.526145 20234 leader_election.cc:288] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:16.526221 20242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:16.526420 20242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [term 1 LEADER]: Becoming Leader. State: Replica: 4793cdef8b514193ace87997a9ac7755, State: Running, Role: LEADER
I20260809 03:30:16.526717 20242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } }
I20260809 03:30:16.527324 20234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:16.527858 20247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4793cdef8b514193ace87997a9ac7755. Latest consensus state: current_term: 1 leader_uuid: "4793cdef8b514193ace87997a9ac7755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } } }
I20260809 03:30:16.527838 20244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4793cdef8b514193ace87997a9ac7755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4793cdef8b514193ace87997a9ac7755" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46253 } } }
I20260809 03:30:16.527973 20244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:16.527973 20247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:16.528683 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:16.516607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.15.222.126:46253)
0809 03:30:16.517024 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:16.517039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:16.517072 (+ 33us) server_negotiation.cc:407] Connection header received
0809 03:30:16.517681 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:16.517698 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:16.518078 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:16.518380 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:16.519192 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.520146 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.521001 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.521458 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.521612 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:16.523871 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:16.523876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:16.523888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:16.523916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:16.525525 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:16.525974 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:16.525980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:16.525986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:16.526097 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:16.526418 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:16.526421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:16.526422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:16.526922 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:16.527112 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:16.527431 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 03:30:16.527727 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":137,"threads_started":1}
W20260809 03:30:16.532083 20277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:16.532161 20277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:16.532217 20251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:16.533052 20251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:16.538316 20251 catalog_manager.cc:1347] Generated new cluster ID: a316f268cde14161a1c1c3967fe3976a
I20260809 03:30:16.538378 20251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:16.544751 20251 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:16.545881 20251 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:16.566061 20251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4793cdef8b514193ace87997a9ac7755: Generated new TSK 0
I20260809 03:30:16.566869 20251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:16.598179 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:16.706826 20285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:16.707070 20285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:16.707116 20285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:16.711000 20285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:16.711050 20285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:16.711120 20285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:16.716065 20285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:16.717016 20285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:16.718015 20285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:16.727074 20291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:16.727048 20294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:16.727242 20285 server_base.cc:1034] running on GCE node
W20260809 03:30:16.727090 20292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:16.727700 20285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:16.728320 20285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:16.729480 20285 hybrid_clock.cc:648] HybridClock initialized: now 1786246216729447 us; error 43 us; skew 500 ppm
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:16.732129 20285 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:16.732915 20285 webserver.cc:460] Webserver started at http://127.15.222.65:34295/ using document root <none> and password file <none>
I20260809 03:30:16.733431 20285 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:16.733500 20285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:16.733729 20285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:16.735671 20285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "b68e610ae9574df4b21bd55da864d9bc"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "b4b4fcd3c1c882d67aaee00b51b711cb"
server_key_iv: "fd5955e0b852149717c62a2d0388d5c2"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.736199 20285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "b68e610ae9574df4b21bd55da864d9bc"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "b4b4fcd3c1c882d67aaee00b51b711cb"
server_key_iv: "fd5955e0b852149717c62a2d0388d5c2"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.739708 20285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:16.741899 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.743000 20285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:16.743098 20285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "b68e610ae9574df4b21bd55da864d9bc"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "b4b4fcd3c1c882d67aaee00b51b711cb"
server_key_iv: "fd5955e0b852149717c62a2d0388d5c2"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.743183 20285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:16.757370 20285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:16.759819 20285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:16.759989 20285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:16.760509 20285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:16.761286 20285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:16.761333 20285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.761384 20285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:16.761413 20285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.769466 20285 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:38833
I20260809 03:30:16.769493 20497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:38833 every 8 connection(s)
I20260809 03:30:16.770320 20285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:16.773885 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 20285
I20260809 03:30:16.773999 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:16.774305 16249 external_mini_cluster.cc:1428] Setting key 9e9ed6f9ebe2a8fc5084ca217b9d3be1
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:16.784618 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:16.772240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47007 (local address 127.15.222.126:46253)
0809 03:30:16.772398 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:16.772402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:16.773195 (+ 793us) server_negotiation.cc:407] Connection header received
0809 03:30:16.774694 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:16.774697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:16.774757 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:16.774847 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:16.776180 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.776689 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.778171 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.778521 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.778627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:16.781101 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:16.781105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:16.781108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:16.781137 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:16.782642 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:16.783113 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:16.783116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:16.783118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:16.783215 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:16.783523 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:16.783527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:16.783529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:16.783760 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:16.783863 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:16.784322 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 03:30:16.784447 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:16.785351 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:16.772623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46253 (local address 127.15.222.65:47007)
0809 03:30:16.773048 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:16.773091 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:16.774471 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:16.775051 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:16.775061 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:16.775458 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:16.776033 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:16.776055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.776821 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.776825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:16.778062 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:16.778073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.778648 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.778651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:16.778781 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:16.779316 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:16.779332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:16.780931 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:16.782749 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:16.782754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:16.782765 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:16.783022 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:16.783304 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:16.783306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:16.783308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:16.783436 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:16.783849 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:16.783854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:16.784087 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 03:30:16.784243 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 03:30:16.784487 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260809 03:30:16.786576 20498 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46253
I20260809 03:30:16.786806 20498 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:16.787292 20498 heartbeater.cc:511] Master 127.15.222.126:46253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:16.788774 20183 ts_manager.cc:194] Registered new tserver with Master: b68e610ae9574df4b21bd55da864d9bc (127.15.222.65:38833)
I20260809 03:30:16.789983 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:47007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:16.801894 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:16.910874 20503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:16.911111 20503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:16.911155 20503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:16.915086 20503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:16.915141 20503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:16.915206 20503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:16.920356 20503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:16.921339 20503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:16.922355 20503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:16.931756 20511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:16.931756 20513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:16.931805 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:16.931949 20503 server_base.cc:1034] running on GCE node
I20260809 03:30:16.932443 20503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:16.932988 20503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:16.934142 20503 hybrid_clock.cc:648] HybridClock initialized: now 1786246216934122 us; error 33 us; skew 500 ppm
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:16.936703 20503 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:16.937556 20503 webserver.cc:460] Webserver started at http://127.15.222.66:40533/ using document root <none> and password file <none>
I20260809 03:30:16.938102 20503 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:16.938161 20503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:16.938329 20503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:16.940291 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "40948adedeef443997c5f0c4dc2d155b"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "4e6031fef2eae109f2461faf5575a167"
server_key_iv: "cf207980b94db81848f3ec27f6e8f8fd"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.940869 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "40948adedeef443997c5f0c4dc2d155b"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "4e6031fef2eae109f2461faf5575a167"
server_key_iv: "cf207980b94db81848f3ec27f6e8f8fd"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.944602 20503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:16.946959 20530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.948100 20503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:16.948192 20503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "40948adedeef443997c5f0c4dc2d155b"
format_stamp: "Formatted at 2026-08-09 03:30:16 on dist-test-slave-cvwc"
server_key: "4e6031fef2eae109f2461faf5575a167"
server_key_iv: "cf207980b94db81848f3ec27f6e8f8fd"
server_key_version: "encryptionkey@0"
I20260809 03:30:16.948272 20503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:16.978247 20503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:16.981060 20503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:16.981208 20503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:16.981794 20503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:16.982602 20503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:16.982649 20503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.982689 20503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:16.982709 20503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:16.991070 20503 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:32969
I20260809 03:30:16.991096 20709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:32969 every 8 connection(s)
I20260809 03:30:16.991935 20503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:16.997833 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 20503
I20260809 03:30:16.997951 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:16.998243 16249 external_mini_cluster.cc:1428] Setting key 644a1bd4d8c0cb23d86c35857f5f8b4d
Aug 09 03:30:16 dist-test-slave-cvwc krb5kdc[20102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:17.005223 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:16.993692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:38747 (local address 127.15.222.126:46253)
0809 03:30:16.993855 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:16.993860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:16.994675 (+ 815us) server_negotiation.cc:407] Connection header received
0809 03:30:16.995873 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:16.995876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:16.995930 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:16.996000 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:16.997090 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.997621 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.999054 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.999268 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.999348 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:17.001986 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:17.001989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:17.001991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:17.002017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:17.003274 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:17.003754 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:17.003757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:17.003758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:17.003836 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:17.004148 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:17.004151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:17.004152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:17.004364 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:17.004453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:17.004942 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 03:30:17.005061 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 03:30:17.006045 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:16.994100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46253 (local address 127.15.222.66:38747)
0809 03:30:16.994534 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:16.994575 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:16.995693 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:16.996153 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:16.996159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:16.996500 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:16.996976 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:16.996990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.997753 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.997757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:16.998943 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:16.998955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:16.999411 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:16.999415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:16.999600 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:17.000164 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:17.000180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:17.001821 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:17.003379 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:17.003384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:17.003407 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:17.003650 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:17.003915 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:17.003918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:17.003920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:17.004054 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:17.004462 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:17.004468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:17.004702 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 03:30:17.004855 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 03:30:17.005165 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260809 03:30:17.007108 20711 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46253
I20260809 03:30:17.007365 20711 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:17.007942 20711 heartbeater.cc:511] Master 127.15.222.126:46253 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:17.009001 20183 ts_manager.cc:194] Registered new tserver with Master: 40948adedeef443997c5f0c4dc2d155b (127.15.222.66:32969)
I20260809 03:30:17.009574 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:38747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:17.024106 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:17.132493 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:17.132717 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:17.132773 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:17.136747 20718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:17.136799 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:17.136873 20718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:17.141893 20718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:17.142820 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:17.143762 20718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:17.152691 20728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:17.152796 20727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:17.152707 20730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:17.152877 20718 server_base.cc:1034] running on GCE node
I20260809 03:30:17.153334 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:17.153931 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:17.155089 20718 hybrid_clock.cc:648] HybridClock initialized: now 1786246217155069 us; error 28 us; skew 500 ppm
Aug 09 03:30:17 dist-test-slave-cvwc krb5kdc[20102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:17.157653 20718 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:17.158499 20718 webserver.cc:460] Webserver started at http://127.15.222.67:41317/ using document root <none> and password file <none>
I20260809 03:30:17.159044 20718 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:17.159109 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:17.159313 20718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:17.161329 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "9a8a23800ce14197a1e7499ef64fd27f"
format_stamp: "Formatted at 2026-08-09 03:30:17 on dist-test-slave-cvwc"
server_key: "9b0b977f4ebdfce3b91f9ef65b83e711"
server_key_iv: "131e5583dfda2005082cc7d0bcf38100"
server_key_version: "encryptionkey@0"
I20260809 03:30:17.161943 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8a23800ce14197a1e7499ef64fd27f"
format_stamp: "Formatted at 2026-08-09 03:30:17 on dist-test-slave-cvwc"
server_key: "9b0b977f4ebdfce3b91f9ef65b83e711"
server_key_iv: "131e5583dfda2005082cc7d0bcf38100"
server_key_version: "encryptionkey@0"
I20260809 03:30:17.166087 20718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 03:30:17.168496 20744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:17.169642 20718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:17.169765 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "9a8a23800ce14197a1e7499ef64fd27f"
format_stamp: "Formatted at 2026-08-09 03:30:17 on dist-test-slave-cvwc"
server_key: "9b0b977f4ebdfce3b91f9ef65b83e711"
server_key_iv: "131e5583dfda2005082cc7d0bcf38100"
server_key_version: "encryptionkey@0"
I20260809 03:30:17.169860 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:17.188786 20718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:17.191740 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:17.191908 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:17.192449 20718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:17.193217 20718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:17.193264 20718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:17.193315 20718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:17.193343 20718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:17.201537 20718 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:42975
I20260809 03:30:17.201566 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:42975 every 8 connection(s)
I20260809 03:30:17.202375 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:17.209659 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 20718
I20260809 03:30:17.209779 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:17.210060 16249 external_mini_cluster.cc:1428] Setting key b121bd556497d6c99335b4dc71a9cd3b
Aug 09 03:30:17 dist-test-slave-cvwc krb5kdc[20102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:17.214790 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:17.204040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:35687 (local address 127.15.222.126:46253)
0809 03:30:17.204153 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:17.204156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:17.204902 (+ 746us) server_negotiation.cc:407] Connection header received
0809 03:30:17.206190 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:17.206194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:17.206254 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:17.206324 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:17.207430 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:17.207944 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:17.209036 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:17.209250 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:17.209335 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:17.211926 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:17.211930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:17.211933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:17.211961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:17.213152 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:17.213593 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:17.213595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:17.213596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:17.213663 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:17.213949 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:17.213951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:17.213952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:17.214117 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:17.214192 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:17.214591 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 03:30:17.214676 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:30:17.215516 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:17.204333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46253 (local address 127.15.222.67:35687)
0809 03:30:17.204751 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:17.204791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:17.205981 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:17.206499 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:17.206506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:17.206829 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:17.207278 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:17.207292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:17.208063 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:17.208065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:17.208918 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:17.208928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:17.209381 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:17.209385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:17.209545 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:17.210089 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:17.210109 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:17.211778 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:17.213256 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:17.213260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:17.213271 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:17.213523 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:17.213739 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:17.213742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:17.213744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:17.213878 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:17.214190 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:17.214194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:17.214413 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 03:30:17.214549 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 03:30:17.214777 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260809 03:30:17.216483 20929 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46253
I20260809 03:30:17.216674 20929 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:17.217092 20929 heartbeater.cc:511] Master 127.15.222.126:46253 requested a full tablet report, sending...
I20260809 03:30:17.217993 20183 ts_manager.cc:194] Registered new tserver with Master: 9a8a23800ce14197a1e7499ef64fd27f (127.15.222.67:42975)
I20260809 03:30:17.218510 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:35687
I20260809 03:30:17.223492 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:30:17.237764 20239 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:17.236178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.15.222.126:46253)
0809 03:30:17.236302 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:17.236307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:17.236480 (+ 173us) server_negotiation.cc:407] Connection header received
0809 03:30:17.236662 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:17.236666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:17.236714 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:17.236783 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:17.237607 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60578: BlockingRecv error: recv got EOF from 127.0.0.1:60578 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:30:17.238924 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 20285
I20260809 03:30:17.251423 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 20503
I20260809 03:30:17.261624 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 20718
I20260809 03:30:17.272411 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 20122
2026-08-09T03:30:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2195 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:17 dist-test-slave-cvwc krb5kdc[20946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:17 dist-test-slave-cvwc krb5kdc[20946](info): set up 2 sockets
Aug 09 03:30:17 dist-test-slave-cvwc krb5kdc[20946](info): commencing operation
krb5kdc: starting...
W20260809 03:30:18.407975 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:18.524271 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35217
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:18.632194 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:18.632432 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:18.632484 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:18.636404 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:18.636454 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:18.636479 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:18.636499 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:18.636520 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:18.641564 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:18.642499 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:18.643442 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:18.651477 20985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.651484 20988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:18.651535 20986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:18.651671 20974 server_base.cc:1034] running on GCE node
I20260809 03:30:18.652272 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:18.653271 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:18.654421 20974 hybrid_clock.cc:648] HybridClock initialized: now 1786246218654409 us; error 23 us; skew 500 ppm
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:18.660445 20974 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:18.661338 20974 webserver.cc:460] Webserver started at http://127.15.222.126:36057/ using document root <none> and password file <none>
I20260809 03:30:18.661906 20974 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:18.661968 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:18.662174 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:18.664355 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "7b8fc6c6e4dc4c868b3a70b641627d83"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-cvwc"
server_key: "b70b79b4a0b93fd876f88434449ad336"
server_key_iv: "c9178b1914951c8074c9b6f7f4d62aa2"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.664934 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "7b8fc6c6e4dc4c868b3a70b641627d83"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-cvwc"
server_key: "b70b79b4a0b93fd876f88434449ad336"
server_key_iv: "c9178b1914951c8074c9b6f7f4d62aa2"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.668669 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:18.671051 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:18.672269 20974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:18.672405 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "7b8fc6c6e4dc4c868b3a70b641627d83"
format_stamp: "Formatted at 2026-08-09 03:30:18 on dist-test-slave-cvwc"
server_key: "b70b79b4a0b93fd876f88434449ad336"
server_key_iv: "c9178b1914951c8074c9b6f7f4d62aa2"
server_key_version: "encryptionkey@0"
I20260809 03:30:18.672503 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:18.687510 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:18.690243 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:18.690423 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:18.697233 20974 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:35217
I20260809 03:30:18.697499 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:35217 every 8 connection(s)
I20260809 03:30:18.698253 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:18.699541 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 20974
I20260809 03:30:18.699656 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:18.699962 16249 external_mini_cluster.cc:1428] Setting key 9d21539e8a9315f25cd2ae1e6eb0f91c
I20260809 03:30:18.701545 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:18.707875 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83: Bootstrap starting.
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:18.710179 21094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:18.710855 21094 log.cc:824] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:18.712672 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83: No bootstrap required, opened a new log
I20260809 03:30:18.713212 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.701218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.15.222.126:35217)
0809 03:30:18.701629 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.701649 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.701693 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:30:18.702358 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.702375 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.702692 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.702972 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.703853 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.704808 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.705917 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.706352 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.706460 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.709031 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.709036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.709048 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:18.709075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.710629 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.711029 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.711034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.711038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.711146 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:18.711390 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:18.711393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:18.711395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:18.711761 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.711952 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.712210 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.712471 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260809 03:30:18.716024 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:18.716539 21094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } }
I20260809 03:30:18.716746 21094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:18.716811 21094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8fc6c6e4dc4c868b3a70b641627d83, State: Initialized, Role: FOLLOWER
I20260809 03:30:18.717229 21094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } }
I20260809 03:30:18.717360 21094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:18.717422 21094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:18.717514 21094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:18.718350 21094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } }
I20260809 03:30:18.718675 21094 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8fc6c6e4dc4c868b3a70b641627d83; no voters:
I20260809 03:30:18.718964 21094 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:18.719050 21108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:18.719226 21108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8fc6c6e4dc4c868b3a70b641627d83, State: Running, Role: LEADER
I20260809 03:30:18.719578 21108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } }
I20260809 03:30:18.720185 21094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:18.720718 21108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8fc6c6e4dc4c868b3a70b641627d83. Latest consensus state: current_term: 1 leader_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } } }
I20260809 03:30:18.720916 21108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:18.721120 21109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc6c6e4dc4c868b3a70b641627d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35217 } } }
I20260809 03:30:18.721294 21109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:18.721334 21122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:18.724427 21122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:18.730561 21122 catalog_manager.cc:1347] Generated new cluster ID: de49d994280f4e6f9ebb5078d2ec8131
I20260809 03:30:18.730610 21122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:18.737764 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:18.732802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.15.222.126:35217)
0809 03:30:18.732919 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:18.732923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:18.732938 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:18.733017 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:18.733019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:18.733060 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:18.733134 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:18.733686 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.734470 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.735610 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:18.735955 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:18.736034 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:18.736866 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:18.736869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:18.736872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:18.737342 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:18.737372 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:18.737391 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:18.737465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:18.737560 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:30:18.737613 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:18.751129 21122 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:18.752277 21122 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:18.759063 21122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8fc6c6e4dc4c868b3a70b641627d83: Generated new TSK 0
I20260809 03:30:18.759676 21122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:18.765452 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 20974
2026-08-09T03:30:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1494 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[21155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[21155](info): set up 2 sockets
Aug 09 03:30:18 dist-test-slave-cvwc krb5kdc[21155](info): commencing operation
krb5kdc: starting...
W20260809 03:30:19.906702 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:19 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:20.024626 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35407
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:35407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:20.133539 21175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:20.133819 21175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:20.133868 21175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:20.137705 21175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:20.137777 21175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:20.137797 21175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:20.137825 21175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:20.137851 21175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:20.142922 21175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:35407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:20.143833 21175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:20.144766 21175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:20.152688 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.152904 21175 server_base.cc:1034] running on GCE node
W20260809 03:30:20.152691 21184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.152678 21192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.153648 21175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:20.154708 21175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:20.155901 21175 hybrid_clock.cc:648] HybridClock initialized: now 1786246220155872 us; error 39 us; skew 500 ppm
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:20.158703 21175 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:20.159592 21175 webserver.cc:460] Webserver started at http://127.15.222.126:45283/ using document root <none> and password file <none>
I20260809 03:30:20.160111 21175 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:20.160176 21175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:20.160388 21175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:20.162406 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "e99c20eca1b241daaa6f62d56ac8702d"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "6324a35e57689f5be5690165fd1368a8"
server_key_iv: "80b274f359e0ca785fa5f5bde4314c73"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.162971 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "e99c20eca1b241daaa6f62d56ac8702d"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "6324a35e57689f5be5690165fd1368a8"
server_key_iv: "80b274f359e0ca785fa5f5bde4314c73"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.166505 21175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:20.168771 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.169981 21175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:20.170097 21175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e99c20eca1b241daaa6f62d56ac8702d"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "6324a35e57689f5be5690165fd1368a8"
server_key_iv: "80b274f359e0ca785fa5f5bde4314c73"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.170195 21175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:20.189229 21175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:20.192034 21175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:20.192231 21175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:20.199304 21175 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:35407
I20260809 03:30:20.199563 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:35407 every 8 connection(s)
I20260809 03:30:20.200371 21175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:20.203111 21289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:20.208359 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d: Bootstrap starting.
I20260809 03:30:20.209844 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 21175
I20260809 03:30:20.209968 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:20.210237 16249 external_mini_cluster.cc:1428] Setting key 490e89747d42b571cf432b4fd7394282
I20260809 03:30:20.210747 21289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:20.211537 21289 log.cc:824] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:20.213469 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d: No bootstrap required, opened a new log
I20260809 03:30:20.217386 21289 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } }
I20260809 03:30:20.217658 21289 raft_consensus.cc:374] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:20.217737 21289 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99c20eca1b241daaa6f62d56ac8702d, State: Initialized, Role: FOLLOWER
I20260809 03:30:20.218243 21289 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } }
I20260809 03:30:20.218400 21289 raft_consensus.cc:388] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:20.218487 21289 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:20.218591 21289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 0 FOLLOWER]: Advancing to term 1
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:20.219626 21289 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } }
I20260809 03:30:20.219957 21289 leader_election.cc:302] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e99c20eca1b241daaa6f62d56ac8702d; no voters:
I20260809 03:30:20.220220 21289 leader_election.cc:288] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:20.220348 21297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:20.220559 21297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [term 1 LEADER]: Becoming Leader. State: Replica: e99c20eca1b241daaa6f62d56ac8702d, State: Running, Role: LEADER
I20260809 03:30:20.220880 21297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } }
I20260809 03:30:20.221467 21289 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:20.222565 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e99c20eca1b241daaa6f62d56ac8702d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } } }
I20260809 03:30:20.222728 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:20.222566 21299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99c20eca1b241daaa6f62d56ac8702d. Latest consensus state: current_term: 1 leader_uuid: "e99c20eca1b241daaa6f62d56ac8702d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99c20eca1b241daaa6f62d56ac8702d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35407 } } }
I20260809 03:30:20.222975 21299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:20.223876 21313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:20.225423 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.211316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.15.222.126:35407)
0809 03:30:20.211736 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.211754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.211790 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:30:20.212559 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.212578 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.212899 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.213174 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.214098 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.215087 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.216266 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.216742 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.216866 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.219717 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.219724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.219741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.219775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.221852 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.222371 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.222378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.222385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.222547 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.222857 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.222860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.222862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.223419 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.223602 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.224281 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.224586 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260809 03:30:20.226902 21313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:20.231984 21313 catalog_manager.cc:1347] Generated new cluster ID: 2ecf9666b2f748189b0f3a2dfe4c49f4
I20260809 03:30:20.232048 21313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:20.248095 21313 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:20.249224 21313 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:20.255865 21313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e99c20eca1b241daaa6f62d56ac8702d: Generated new TSK 0
I20260809 03:30:20.256577 21313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:20.288589 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:20.396785 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:20.397048 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:20.397109 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:20.401069 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:20.401129 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:20.401206 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:20.406260 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:20.407202 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:20.408162 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:20.417197 21338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.417270 21331 server_base.cc:1034] running on GCE node
W20260809 03:30:20.417279 21337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.417434 21340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.417843 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:20.418448 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:20.419610 21331 hybrid_clock.cc:648] HybridClock initialized: now 1786246220419582 us; error 39 us; skew 500 ppm
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:20.422297 21331 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:20.423179 21331 webserver.cc:460] Webserver started at http://127.15.222.65:42181/ using document root <none> and password file <none>
I20260809 03:30:20.423727 21331 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:20.423794 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:20.424005 21331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:20.426281 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "cd56de33694c477792dd6a172b06d109"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "85828051e580307a70a14ac1c6788819"
server_key_iv: "be8bde4610b000f981e9b5d15229d91e"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.426815 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "cd56de33694c477792dd6a172b06d109"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "85828051e580307a70a14ac1c6788819"
server_key_iv: "be8bde4610b000f981e9b5d15229d91e"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.430492 21331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:20.432614 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.433691 21331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:20.433825 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "cd56de33694c477792dd6a172b06d109"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "85828051e580307a70a14ac1c6788819"
server_key_iv: "be8bde4610b000f981e9b5d15229d91e"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.433924 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:20.444913 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:20.447343 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:20.447510 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:20.448073 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:20.448787 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:20.448832 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.448868 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:20.448889 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.456688 21331 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:46783
I20260809 03:30:20.456713 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:46783 every 8 connection(s)
I20260809 03:30:20.457469 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:20.464335 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 21331
I20260809 03:30:20.464459 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:20.464771 16249 external_mini_cluster.cc:1428] Setting key afa8aa7bcfaa1a505a8b60ebec52a233
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:20.470180 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.459054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:53781 (local address 127.15.222.126:35407)
0809 03:30:20.459172 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.459176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.459916 (+ 740us) server_negotiation.cc:407] Connection header received
0809 03:30:20.461267 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.461271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.461331 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.461401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.462546 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.463092 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.464166 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.464404 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.464495 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.466810 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.466814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.466817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.466844 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.468207 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.468730 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.468732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.468734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.468801 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.469136 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.469138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.469139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.469294 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.469360 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.469897 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.470021 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:20.470908 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.459349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35407 (local address 127.15.222.65:53781)
0809 03:30:20.459774 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:20.459813 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:20.461063 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:20.461584 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:20.461590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:20.461955 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:20.462405 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.462418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.463208 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.463210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.464046 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.464055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.464546 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.464551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.464748 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.465278 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:20.465294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:20.466656 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:20.468312 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.468318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.468333 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.468649 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.468868 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.468871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.468873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.469051 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.469379 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:20.469385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:20.469630 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 03:30:20.469823 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 03:30:20.470129 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260809 03:30:20.472056 21548 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35407
I20260809 03:30:20.472306 21548 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:20.472853 21548 heartbeater.cc:511] Master 127.15.222.126:35407 requested a full tablet report, sending...
I20260809 03:30:20.474469 21229 ts_manager.cc:194] Registered new tserver with Master: cd56de33694c477792dd6a172b06d109 (127.15.222.65:46783)
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:20.475584 21229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:53781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:20.491971 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:20.603323 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:20.603570 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:20.603611 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:20.607466 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:20.607510 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:20.607563 21560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:20.612505 21560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:20.613392 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:20.614439 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:20.623767 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.623773 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.623775 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.624022 21560 server_base.cc:1034] running on GCE node
I20260809 03:30:20.624490 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:20.625123 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:20.626267 21560 hybrid_clock.cc:648] HybridClock initialized: now 1786246220626242 us; error 39 us; skew 500 ppm
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:20.628914 21560 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:20.629766 21560 webserver.cc:460] Webserver started at http://127.15.222.66:44835/ using document root <none> and password file <none>
I20260809 03:30:20.630298 21560 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:20.630357 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:20.630530 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:20.632490 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "a00d46e122f24e7e863c7486f432ed78"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "4458e32a1fc89619c51ff6dd8846edf0"
server_key_iv: "81da1acb9b9f6c8c888c7bbf7ee20254"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.633008 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "a00d46e122f24e7e863c7486f432ed78"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "4458e32a1fc89619c51ff6dd8846edf0"
server_key_iv: "81da1acb9b9f6c8c888c7bbf7ee20254"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.636497 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:30:20.638859 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.640021 21560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:20.640125 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "a00d46e122f24e7e863c7486f432ed78"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "4458e32a1fc89619c51ff6dd8846edf0"
server_key_iv: "81da1acb9b9f6c8c888c7bbf7ee20254"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.640211 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:20.654431 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:20.657322 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:20.657516 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:20.658120 21560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:20.658921 21560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:20.658968 21560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.659018 21560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:20.659044 21560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.667076 21560 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:40137
I20260809 03:30:20.667127 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:40137 every 8 connection(s)
I20260809 03:30:20.667936 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:20.678117 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 21560
I20260809 03:30:20.678258 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:20.678545 16249 external_mini_cluster.cc:1428] Setting key 6e72c90035e2bc33ef35dcf7a26cc7da
I20260809 03:30:20.681187 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.669795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40187 (local address 127.15.222.126:35407)
0809 03:30:20.669939 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.669943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.670778 (+ 835us) server_negotiation.cc:407] Connection header received
0809 03:30:20.672071 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.672074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.672122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.672194 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.673282 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.673972 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.675158 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.675386 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.675481 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.677795 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.677799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.677802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.677830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.679277 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.679742 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.679745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.679747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.679823 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.680140 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.680144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.680146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.680322 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.680408 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.680903 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.681026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:20.681995 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.670183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35407 (local address 127.15.222.66:40187)
0809 03:30:20.670634 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:20.670674 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:20.671869 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:20.672353 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:20.672359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:20.672702 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:20.673175 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.673188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.674111 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.674113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.675040 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.675049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.675535 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.675540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.675704 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.676240 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:20.676257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:20.677614 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:20.679379 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.679385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.679408 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.679648 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.679904 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.679907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.679909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.680054 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.680426 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:20.680433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:20.680668 (+ 235us) client_negotiation.cc:769] Sending connection context
0809 03:30:20.680833 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 03:30:20.681146 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260809 03:30:20.683151 21782 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35407
I20260809 03:30:20.683379 21782 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:20.683848 21782 heartbeater.cc:511] Master 127.15.222.126:35407 requested a full tablet report, sending...
I20260809 03:30:20.684893 21229 ts_manager.cc:194] Registered new tserver with Master: a00d46e122f24e7e863c7486f432ed78 (127.15.222.66:40137)
I20260809 03:30:20.685461 21229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:40187
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:20.708050 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:20.816829 21788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:20.817070 21788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:20.817111 21788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:20.821187 21788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:20.821236 21788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:20.821303 21788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:20.826493 21788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:20.827422 21788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:20.828363 21788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:20.837366 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:20.837386 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.837574 21788 server_base.cc:1034] running on GCE node
W20260809 03:30:20.837795 21799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:20.838201 21788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:20.838816 21788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:20.839975 21788 hybrid_clock.cc:648] HybridClock initialized: now 1786246220839958 us; error 25 us; skew 500 ppm
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:20.842562 21788 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:20.843410 21788 webserver.cc:460] Webserver started at http://127.15.222.67:39279/ using document root <none> and password file <none>
I20260809 03:30:20.843969 21788 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:20.844031 21788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:20.844233 21788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:20.846354 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "13f9a74d558d4e789449436c53214997"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "7f46d0571bd48320e15b9038cf4f91a0"
server_key_iv: "f0bcde8a654e23fd915f4ecd9abf27ea"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.847033 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "13f9a74d558d4e789449436c53214997"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "7f46d0571bd48320e15b9038cf4f91a0"
server_key_iv: "f0bcde8a654e23fd915f4ecd9abf27ea"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.855089 21788 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260809 03:30:20.857853 21813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.859020 21788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 03:30:20.859124 21788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "13f9a74d558d4e789449436c53214997"
format_stamp: "Formatted at 2026-08-09 03:30:20 on dist-test-slave-cvwc"
server_key: "7f46d0571bd48320e15b9038cf4f91a0"
server_key_iv: "f0bcde8a654e23fd915f4ecd9abf27ea"
server_key_version: "encryptionkey@0"
I20260809 03:30:20.859222 21788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:20.880244 21788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:20.882915 21788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:20.883086 21788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:20.883674 21788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:20.884487 21788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:20.884536 21788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.884588 21788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:20.884616 21788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:20.892642 21788 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:44773
I20260809 03:30:20.892674 22006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:44773 every 8 connection(s)
I20260809 03:30:20.893481 21788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:20.894119 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 21788
I20260809 03:30:20.894224 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:20.894500 16249 external_mini_cluster.cc:1428] Setting key 556cfa7d31fea90acb71ba12e565bb8a
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:20.905903 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.895253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:44499 (local address 127.15.222.126:35407)
0809 03:30:20.895387 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.895399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.896155 (+ 756us) server_negotiation.cc:407] Connection header received
0809 03:30:20.897612 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.897615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.897668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.897752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.898885 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.899406 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.900480 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.900690 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.900773 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.902991 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.902994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.902996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.903018 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.904078 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.904523 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.904526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.904527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.904599 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.904890 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.904894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.904895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.905065 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.905148 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.905601 (+ 453us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.905745 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:20.906608 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.895593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35407 (local address 127.15.222.67:44499)
0809 03:30:20.896015 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:20.896056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:20.897424 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:20.897936 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:20.897943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:20.898280 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:20.898755 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.898769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.899518 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.899521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.900366 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:20.900376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.900817 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.900822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:20.900984 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.901443 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:20.901458 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:20.902830 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:20.904179 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.904185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.904199 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.904433 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.904678 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:20.904681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:20.904683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:20.904807 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:20.905163 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:20.905169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:20.905385 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 03:30:20.905544 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 03:30:20.905863 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260809 03:30:20.907545 22007 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35407
I20260809 03:30:20.907735 22007 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:20.908159 22007 heartbeater.cc:511] Master 127.15.222.126:35407 requested a full tablet report, sending...
I20260809 03:30:20.909161 21229 ts_manager.cc:194] Registered new tserver with Master: 13f9a74d558d4e789449436c53214997 (127.15.222.67:44773)
I20260809 03:30:20.909523 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:20.909859 21229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:44499
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 03:30:20 dist-test-slave-cvwc krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246220, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:20.939564 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:20.931333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.15.222.126:35407)
0809 03:30:20.931457 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:20.931460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:20.931994 (+ 534us) server_negotiation.cc:407] Connection header received
0809 03:30:20.932181 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:20.932184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:20.932242 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:20.932329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:20.933080 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.933611 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.934418 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:20.934631 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:20.934706 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:20.936894 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:20.936898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:20.936901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:20.936926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:20.938241 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.938591 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.938593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.938595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.938662 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:20.938841 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:20.938843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:20.938844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:20.939051 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:20.939147 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:20.939306 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 03:30:20.939403 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 03:30:20.939939 21229 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52846
I20260809 03:30:20.940698 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 21331
I20260809 03:30:20.953593 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 21560
I20260809 03:30:20.961647 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 21788
I20260809 03:30:20.984798 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 21175
2026-08-09T03:30:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2217 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:21 dist-test-slave-cvwc krb5kdc[22022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:21 dist-test-slave-cvwc krb5kdc[22022](info): set up 2 sockets
Aug 09 03:30:21 dist-test-slave-cvwc krb5kdc[22022](info): commencing operation
krb5kdc: starting...
W20260809 03:30:22.120767 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:22.233784 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44435
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:44435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.340286 22050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.340530 22050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.340569 22050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.344412 22050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:22.344458 22050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.344475 22050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.344492 22050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:22.344509 22050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:22.349484 22050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:44435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:22.350443 22050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.351397 22050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:22.359483 22050 server_base.cc:1034] running on GCE node
W20260809 03:30:22.359411 22060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.359380 22061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.359390 22064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.360172 22050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.361209 22050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.362363 22050 hybrid_clock.cc:648] HybridClock initialized: now 1786246222362336 us; error 38 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.364907 22050 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:22.365705 22050 webserver.cc:460] Webserver started at http://127.15.222.126:43545/ using document root <none> and password file <none>
I20260809 03:30:22.366225 22050 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.366282 22050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.366452 22050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.368333 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "a060fbb8eebc4bbe98d13e8de28e3742"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "8b3f70f9ab412e5d9f20572c5da3d00d"
server_key_iv: "9a7e46131a2e217708cb3091cb7f0ca3"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.368835 22050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "a060fbb8eebc4bbe98d13e8de28e3742"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "8b3f70f9ab412e5d9f20572c5da3d00d"
server_key_iv: "9a7e46131a2e217708cb3091cb7f0ca3"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.372282 22050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:30:22.374480 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.375598 22050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:22.375700 22050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "a060fbb8eebc4bbe98d13e8de28e3742"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "8b3f70f9ab412e5d9f20572c5da3d00d"
server_key_iv: "9a7e46131a2e217708cb3091cb7f0ca3"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.375794 22050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.390357 22050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.392887 22050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.393049 22050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.399544 22050 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:44435
I20260809 03:30:22.399557 22186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:44435 every 8 connection(s)
I20260809 03:30:22.400429 22050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:22.403069 22187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:22.408298 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: Bootstrap starting.
I20260809 03:30:22.408648 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 22050
I20260809 03:30:22.408761 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:22.408991 16249 external_mini_cluster.cc:1428] Setting key a1155ad3816b0477b50a7d067789fa27
I20260809 03:30:22.410750 22187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:22.411461 22187 log.cc:824] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:22.413379 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: No bootstrap required, opened a new log
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:22.417524 22187 raft_consensus.cc:348] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } }
I20260809 03:30:22.417773 22187 raft_consensus.cc:374] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:22.417868 22187 raft_consensus.cc:729] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a060fbb8eebc4bbe98d13e8de28e3742, State: Initialized, Role: FOLLOWER
I20260809 03:30:22.418303 22187 consensus_queue.cc:260] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } }
I20260809 03:30:22.418443 22187 raft_consensus.cc:388] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:22.418520 22187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:22.418620 22187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:22.419412 22187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } }
I20260809 03:30:22.419729 22187 leader_election.cc:302] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a060fbb8eebc4bbe98d13e8de28e3742; no voters:
I20260809 03:30:22.420013 22187 leader_election.cc:288] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:22.420090 22197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:22.420293 22197 raft_consensus.cc:686] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [term 1 LEADER]: Becoming Leader. State: Replica: a060fbb8eebc4bbe98d13e8de28e3742, State: Running, Role: LEADER
I20260809 03:30:22.420635 22197 consensus_queue.cc:237] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } }
I20260809 03:30:22.421267 22187 sys_catalog.cc:564] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:22.422029 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a060fbb8eebc4bbe98d13e8de28e3742. Latest consensus state: current_term: 1 leader_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } } }
I20260809 03:30:22.422741 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:22.422909 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.410209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.15.222.126:44435)
0809 03:30:22.410612 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.410626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.410658 (+ 32us) server_negotiation.cc:407] Connection header received
0809 03:30:22.411300 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.411318 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.411642 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.411972 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.412742 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.413741 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.414864 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.415306 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.415412 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.418016 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.418021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.418034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.418060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.419745 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.420190 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.420196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.420202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.420348 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.420651 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.420655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.420657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.421065 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.421248 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.421626 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.421932 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260809 03:30:22.422636 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a060fbb8eebc4bbe98d13e8de28e3742" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44435 } } }
I20260809 03:30:22.423022 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742 [sys.catalog]: This master's current role is: LEADER
W20260809 03:30:22.426340 22222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:22.426420 22222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:22.426501 22214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:22.427337 22214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:22.432518 22214 catalog_manager.cc:1347] Generated new cluster ID: 597d24fbb21f42ae95d70dd1b330e055
I20260809 03:30:22.432580 22214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:22.443545 22214 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:22.444388 22214 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:22.452046 22214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a060fbb8eebc4bbe98d13e8de28e3742: Generated new TSK 0
I20260809 03:30:22.452883 22214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:22.491164 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.603446 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.603684 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.603739 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.607618 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.607667 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.607743 22232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:22.612756 22232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:22.613693 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.614657 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:22.623767 22248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.623778 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.623804 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.624073 22232 server_base.cc:1034] running on GCE node
I20260809 03:30:22.624460 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.625034 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.626194 22232 hybrid_clock.cc:648] HybridClock initialized: now 1786246222626171 us; error 33 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.628708 22232 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:22.629544 22232 webserver.cc:460] Webserver started at http://127.15.222.65:44837/ using document root <none> and password file <none>
I20260809 03:30:22.630101 22232 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.630168 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.630368 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.632444 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "c9b967c50f82410daa86d166e3966901"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "bef06f3546c09afcd600be6eeab22005"
server_key_iv: "f49460624aff88d8c212e3d9d08a21cf"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.633004 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "c9b967c50f82410daa86d166e3966901"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "bef06f3546c09afcd600be6eeab22005"
server_key_iv: "f49460624aff88d8c212e3d9d08a21cf"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.636533 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:22.638866 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.639995 22232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:22.640101 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "c9b967c50f82410daa86d166e3966901"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "bef06f3546c09afcd600be6eeab22005"
server_key_iv: "f49460624aff88d8c212e3d9d08a21cf"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.640192 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.653996 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.656630 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.656803 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.657395 22232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:22.658274 22232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:22.658324 22232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.658377 22232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:22.658406 22232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.666707 22232 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:45677
I20260809 03:30:22.666731 22443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:45677 every 8 connection(s)
I20260809 03:30:22.667627 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:22.677884 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 22232
I20260809 03:30:22.677991 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:22.678311 16249 external_mini_cluster.cc:1428] Setting key 94da451f6ceab0d6fc2a9444c0980a2f
I20260809 03:30:22.680908 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.669402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42839 (local address 127.15.222.126:44435)
0809 03:30:22.669529 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.669532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.670367 (+ 835us) server_negotiation.cc:407] Connection header received
0809 03:30:22.671737 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.671741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.671820 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.671903 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.673040 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.673558 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.674691 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.674927 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.675018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.677328 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.677331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.677333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.677359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.678898 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.679404 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.679407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.679409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.679498 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.679838 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.679842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.679843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.680032 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.680120 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.680619 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.680738 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:22.681680 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.669781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44435 (local address 127.15.222.65:42839)
0809 03:30:22.670226 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.670268 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.671534 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.672074 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.672081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.672433 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:22.672896 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.672909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.673675 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.673678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.674574 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.674583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.675066 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.675070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.675232 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.675729 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:22.675743 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:22.677171 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:22.679001 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.679007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.679021 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.679307 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.679590 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.679593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.679595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.679731 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.680132 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:22.680138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:22.680389 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.680559 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.680870 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260809 03:30:22.682828 22444 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44435
I20260809 03:30:22.683043 22444 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:22.683481 22444 heartbeater.cc:511] Master 127.15.222.126:44435 requested a full tablet report, sending...
I20260809 03:30:22.685105 22118 ts_manager.cc:194] Registered new tserver with Master: c9b967c50f82410daa86d166e3966901 (127.15.222.65:45677)
I20260809 03:30:22.686720 22118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:42839
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:22.706251 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:22.815594 22450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:22.815824 22450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:22.815867 22450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:22.819774 22450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:22.819823 22450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:22.819890 22450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:22.824990 22450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:22.825976 22450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:22.826956 22450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:22.836024 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.836032 22466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:22.836032 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:22.836408 22450 server_base.cc:1034] running on GCE node
I20260809 03:30:22.836774 22450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:22.837337 22450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:22.838500 22450 hybrid_clock.cc:648] HybridClock initialized: now 1786246222838482 us; error 26 us; skew 500 ppm
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:22.841051 22450 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:22.841892 22450 webserver.cc:460] Webserver started at http://127.15.222.66:37725/ using document root <none> and password file <none>
I20260809 03:30:22.842406 22450 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:22.842465 22450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:22.842659 22450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:22.844686 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "ae00d68e7de6477e9066ad5a09ac74f6"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "eb68eed1bf8dd85a8769dde22a9dca3e"
server_key_iv: "d69339b8b5dd7aaa5b6dbd1b476fe856"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.845278 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "ae00d68e7de6477e9066ad5a09ac74f6"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "eb68eed1bf8dd85a8769dde22a9dca3e"
server_key_iv: "d69339b8b5dd7aaa5b6dbd1b476fe856"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.849862 22450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 03:30:22.852324 22476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.853506 22450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:22.853603 22450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "ae00d68e7de6477e9066ad5a09ac74f6"
format_stamp: "Formatted at 2026-08-09 03:30:22 on dist-test-slave-cvwc"
server_key: "eb68eed1bf8dd85a8769dde22a9dca3e"
server_key_iv: "d69339b8b5dd7aaa5b6dbd1b476fe856"
server_key_version: "encryptionkey@0"
I20260809 03:30:22.853699 22450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:22.906255 22450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:22.909502 22450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:22.909683 22450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:22.910308 22450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:22.911135 22450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:22.911181 22450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.911227 22450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:22.911255 22450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:22.919540 22450 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:37453
I20260809 03:30:22.919575 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:37453 every 8 connection(s)
I20260809 03:30:22.920414 22450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:22.922410 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 22450
I20260809 03:30:22.922514 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:22.922767 16249 external_mini_cluster.cc:1428] Setting key c142c4fb95a7f270ad43f7c800b7e014
Aug 09 03:30:22 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:22.933992 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.922166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:50275 (local address 127.15.222.126:44435)
0809 03:30:22.922289 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:22.922293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:22.923877 (+ 1584us) server_negotiation.cc:407] Connection header received
0809 03:30:22.924789 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:22.924792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:22.924841 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:22.924931 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:22.926097 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.926632 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.928146 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.928345 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.928411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.930816 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:22.930819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:22.930821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:22.930845 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:22.932154 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.932618 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.932621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.932622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.932698 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:22.933069 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:22.933071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:22.933072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:22.933236 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:22.933307 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:22.933773 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 03:30:22.933870 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:22.934794 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:22.922641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44435 (local address 127.15.222.66:50275)
0809 03:30:22.923146 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:22.923202 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:22.924585 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:22.925072 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:22.925080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:22.925423 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:22.925936 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.925955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.926747 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.926750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.928038 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:22.928049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:22.928475 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:22.928479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:22.928667 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:22.929273 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:22.929290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:22.930660 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:22.932262 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.932267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.932282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.932525 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.932796 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:22.932799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:22.932801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:22.932994 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:22.933308 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:22.933313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:22.933553 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 03:30:22.933708 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 03:30:22.933999 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
I20260809 03:30:22.935889 22660 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44435
I20260809 03:30:22.936156 22660 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:22.936726 22660 heartbeater.cc:511] Master 127.15.222.126:44435 requested a full tablet report, sending...
I20260809 03:30:22.937815 22118 ts_manager.cc:194] Registered new tserver with Master: ae00d68e7de6477e9066ad5a09ac74f6 (127.15.222.66:37453)
I20260809 03:30:22.938366 22118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:50275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:22.948303 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:23.057605 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:23.057881 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:23.057926 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:23.061810 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:23.061861 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:23.061913 22669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:23.066990 22669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:23.067940 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:23.068879 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:23.078009 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.078094 22669 server_base.cc:1034] running on GCE node
W20260809 03:30:23.077999 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:23.078009 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:23.078655 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:23.079288 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:23.080446 22669 hybrid_clock.cc:648] HybridClock initialized: now 1786246223080427 us; error 29 us; skew 500 ppm
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:23.083109 22669 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:23.083976 22669 webserver.cc:460] Webserver started at http://127.15.222.67:32977/ using document root <none> and password file <none>
I20260809 03:30:23.084506 22669 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:23.084573 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:23.084772 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:23.086808 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "d5ef49d99d254f6e904dfb24dc14beb8"
format_stamp: "Formatted at 2026-08-09 03:30:23 on dist-test-slave-cvwc"
server_key: "cad330e17dbdb57b5fe5b13a6b5cf6d4"
server_key_iv: "5c1ef4fb35d6b0e636affae146800bc2"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.087345 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "d5ef49d99d254f6e904dfb24dc14beb8"
format_stamp: "Formatted at 2026-08-09 03:30:23 on dist-test-slave-cvwc"
server_key: "cad330e17dbdb57b5fe5b13a6b5cf6d4"
server_key_iv: "5c1ef4fb35d6b0e636affae146800bc2"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.090847 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:23.093086 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.094283 22669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:23.094386 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d5ef49d99d254f6e904dfb24dc14beb8"
format_stamp: "Formatted at 2026-08-09 03:30:23 on dist-test-slave-cvwc"
server_key: "cad330e17dbdb57b5fe5b13a6b5cf6d4"
server_key_iv: "5c1ef4fb35d6b0e636affae146800bc2"
server_key_version: "encryptionkey@0"
I20260809 03:30:23.094476 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:23.107752 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:23.110630 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:23.110790 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:23.111339 22669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:23.112094 22669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:23.112140 22669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.112190 22669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:23.112218 22669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:23.120393 22669 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:35003
I20260809 03:30:23.120415 22883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:35003 every 8 connection(s)
I20260809 03:30:23.121201 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:23.124145 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 22669
I20260809 03:30:23.124246 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:23.124511 16249 external_mini_cluster.cc:1428] Setting key e0f91acb57979f5175cf9b104176dcfe
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:23.133874 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.122913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:43221 (local address 127.15.222.126:44435)
0809 03:30:23.123052 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.123055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.123853 (+ 798us) server_negotiation.cc:407] Connection header received
0809 03:30:23.125321 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.125324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.125381 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.125459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.126633 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.127149 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.128265 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.128541 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.128633 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.130870 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.130872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.130874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.130900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.132007 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.132455 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.132458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.132460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.132533 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.132832 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.132835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.132837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.133027 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.133112 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.133574 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.133688 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 03:30:23.134631 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.123275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44435 (local address 127.15.222.67:43221)
0809 03:30:23.123709 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:23.123748 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:23.125121 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:23.125631 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:23.125639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:23.126021 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:23.126496 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.126509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.127262 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.127265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.128119 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:23.128129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.128682 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.128686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:23.128848 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.129347 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:23.129362 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:23.130721 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:23.132109 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.132114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.132128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.132360 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.132613 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:23.132616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:23.132618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:23.132744 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:23.133128 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:23.133135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:23.133357 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 03:30:23.133515 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 03:30:23.133847 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260809 03:30:23.135625 22885 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44435
I20260809 03:30:23.135830 22885 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:23.136277 22885 heartbeater.cc:511] Master 127.15.222.126:44435 requested a full tablet report, sending...
I20260809 03:30:23.137240 22118 ts_manager.cc:194] Registered new tserver with Master: d5ef49d99d254f6e904dfb24dc14beb8 (127.15.222.67:35003)
I20260809 03:30:23.137964 22118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:43221
I20260809 03:30:23.138103 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:23.163151 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.154658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.15.222.65:45677)
0809 03:30:23.154962 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.154975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.155000 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:30:23.155049 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.155054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.155233 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.155351 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.155946 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.156766 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.157629 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.158044 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.158129 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.160257 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.160266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.160279 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.160304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.161784 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.162115 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.162120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.162121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.162195 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.162371 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.162373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.162374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.162592 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.162747 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.162872 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.163002 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:23.172824 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:23.165771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.15.222.126:44435)
0809 03:30:23.165914 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:23.165918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:23.165959 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:30:23.166062 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:23.166064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:23.166103 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:23.166169 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:23.166695 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.167181 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.167868 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:23.168083 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:23.168163 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:23.170276 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:23.170279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:23.170281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:23.170305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:23.171543 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.171861 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.171863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.171864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.171934 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:23.172106 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:23.172108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:23.172109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:23.172315 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:23.172411 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:23.172580 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 03:30:23.172667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 03:30:23.173202 22118 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49250
I20260809 03:30:23.173786 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 22232
I20260809 03:30:23.185695 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 22450
I20260809 03:30:23.213310 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 22669
I20260809 03:30:23.222097 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 22050
2026-08-09T03:30:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2239 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22904](info): set up 2 sockets
Aug 09 03:30:23 dist-test-slave-cvwc krb5kdc[22904](info): commencing operation
krb5kdc: starting...
W20260809 03:30:24.361155 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:24 dist-test-slave-cvwc krb5kdc[22904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:24.472329 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44633
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:24.581485 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:24.581789 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:24.581851 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:24.585842 22923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:24.585906 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:24.585924 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:24.585952 22923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:24.585980 22923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:24.591215 22923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:24.592267 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:24.593313 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:24.601490 22941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:24.601500 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:24.601563 22923 server_base.cc:1034] running on GCE node
W20260809 03:30:24.601507 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:24.602344 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:24.603416 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:24.604573 22923 hybrid_clock.cc:648] HybridClock initialized: now 1786246224604551 us; error 31 us; skew 500 ppm
Aug 09 03:30:24 dist-test-slave-cvwc krb5kdc[22904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:24.610332 22923 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:24.611068 22923 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:24.611132 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:24.611325 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:24.613355 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "dba87714909846d4822568098234080f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "6b8c09b1acdc49b649cf8d90d51f68a2"
server_key_iv: "caccf8671cd2ba7f6a5130cc458cc000"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.613940 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "dba87714909846d4822568098234080f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "6b8c09b1acdc49b649cf8d90d51f68a2"
server_key_iv: "caccf8671cd2ba7f6a5130cc458cc000"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.617516 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260809 03:30:24.619894 22949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.621119 22923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:24.621260 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "dba87714909846d4822568098234080f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "6b8c09b1acdc49b649cf8d90d51f68a2"
server_key_iv: "caccf8671cd2ba7f6a5130cc458cc000"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.621356 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:24.648314 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:24.651604 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:24.651789 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:24.658643 22923 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:44633
I20260809 03:30:24.658882 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:44633 every 8 connection(s)
I20260809 03:30:24.659296 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:24.662143 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:24.667567 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: Bootstrap starting.
I20260809 03:30:24.668303 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 22923
I20260809 03:30:24.668431 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:24.668668 16249 external_mini_cluster.cc:1428] Setting key 41a6239b86f6639c63e5a7baff354288
I20260809 03:30:24.670358 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:24.671257 23045 log.cc:824] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:24.673103 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: No bootstrap required, opened a new log
I20260809 03:30:24.676627 23045 raft_consensus.cc:348] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } }
I20260809 03:30:24.676844 23045 raft_consensus.cc:374] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:24.676910 23045 raft_consensus.cc:729] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba87714909846d4822568098234080f, State: Initialized, Role: FOLLOWER
Aug 09 03:30:24 dist-test-slave-cvwc krb5kdc[22904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:24.677338 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } }
I20260809 03:30:24.677476 23045 raft_consensus.cc:388] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:24.677541 23045 raft_consensus.cc:482] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:24.677667 23045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:24.678725 23045 raft_consensus.cc:504] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } }
I20260809 03:30:24.679064 23045 leader_election.cc:302] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dba87714909846d4822568098234080f; no voters:
I20260809 03:30:24.679425 23045 leader_election.cc:288] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:24.679503 23053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:24.679708 23053 raft_consensus.cc:686] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [term 1 LEADER]: Becoming Leader. State: Replica: dba87714909846d4822568098234080f, State: Running, Role: LEADER
I20260809 03:30:24.680111 23053 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } }
I20260809 03:30:24.680833 23045 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:24.681738 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba87714909846d4822568098234080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } } }
I20260809 03:30:24.681874 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:24.682250 23055 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba87714909846d4822568098234080f. Latest consensus state: current_term: 1 leader_uuid: "dba87714909846d4822568098234080f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba87714909846d4822568098234080f" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44633 } } }
I20260809 03:30:24.682401 23055 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:24.683058 23069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:24.683415 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.669847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.15.222.126:44633)
0809 03:30:24.670313 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:24.670333 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:24.670378 (+ 45us) server_negotiation.cc:407] Connection header received
0809 03:30:24.671251 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:24.671276 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:24.671707 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:24.672025 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:24.672790 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.673826 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.674647 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.675127 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.675275 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:24.677643 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:24.677648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:24.677661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:24.677689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:24.679493 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:24.680072 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:24.680079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:24.680084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:24.680229 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:24.680549 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:24.680553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:24.680555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:24.681083 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:24.681289 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:24.682169 (+ 880us) server_negotiation.cc:299] Negotiation successful
0809 03:30:24.682498 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":110,"threads_started":1}
I20260809 03:30:24.685982 23069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:24.691118 23069 catalog_manager.cc:1347] Generated new cluster ID: 01c8ab6d374e4b1fa333d4f273cc4a08
I20260809 03:30:24.691182 23069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:24.705564 23069 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:24.707255 23069 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:24.711675 23069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: Generated new TSK 0
I20260809 03:30:24.712404 23069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:24.744208 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:24.854815 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:24.855064 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:24.855129 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:24.859129 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:24.859189 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:24.859269 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:24.864408 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:24.865406 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:24.866417 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:24.875568 23092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:24.875553 23093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:24.875532 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:24.875628 23084 server_base.cc:1034] running on GCE node
I20260809 03:30:24.876185 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:24.876773 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:24.877923 23084 hybrid_clock.cc:648] HybridClock initialized: now 1786246224877904 us; error 28 us; skew 500 ppm
Aug 09 03:30:24 dist-test-slave-cvwc krb5kdc[22904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:24.880504 23084 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:24.881234 23084 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:24.881299 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:24.881498 23084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:24.883540 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "38cb5833d71743ebb893b0f0e297ce4f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "004eeff3f8791c847d3f6e7a5b00f6b4"
server_key_iv: "eb8a328ab48164d62d27e6f369441285"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.884063 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "38cb5833d71743ebb893b0f0e297ce4f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "004eeff3f8791c847d3f6e7a5b00f6b4"
server_key_iv: "eb8a328ab48164d62d27e6f369441285"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.887530 23084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:24.889786 23114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.890935 23084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:24.891029 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "38cb5833d71743ebb893b0f0e297ce4f"
format_stamp: "Formatted at 2026-08-09 03:30:24 on dist-test-slave-cvwc"
server_key: "004eeff3f8791c847d3f6e7a5b00f6b4"
server_key_iv: "eb8a328ab48164d62d27e6f369441285"
server_key_version: "encryptionkey@0"
I20260809 03:30:24.891119 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:24.907575 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:24.910468 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:24.910630 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:24.911154 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:24.911926 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:24.911971 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.912012 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:24.912041 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:24.920498 23084 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:39303
I20260809 03:30:24.920519 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:39303 every 8 connection(s)
I20260809 03:30:24.921052 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
Aug 09 03:30:24 dist-test-slave-cvwc krb5kdc[22904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246224, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:24.930359 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 23084
I20260809 03:30:24.930490 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:24.930757 16249 external_mini_cluster.cc:1428] Setting key 2a64c5d9d25336ae57154450712adc9e
I20260809 03:30:24.933851 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.922669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:52449 (local address 127.15.222.126:44633)
0809 03:30:24.922791 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:24.922794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:24.923893 (+ 1099us) server_negotiation.cc:407] Connection header received
0809 03:30:24.924788 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:24.924792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:24.924847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:24.924926 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:24.926137 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.926664 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.927778 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.928017 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.928108 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:24.930387 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:24.930390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:24.930392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:24.930418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:24.931867 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:24.932437 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:24.932440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:24.932441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:24.932516 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:24.932847 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:24.932849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:24.932850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:24.933014 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:24.933079 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:24.933612 (+ 533us) server_negotiation.cc:299] Negotiation successful
0809 03:30:24.933702 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:24.934769 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:24.922996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44633 (local address 127.15.222.65:52449)
0809 03:30:24.923752 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:24.923791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:24.924591 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:24.925099 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:24.925105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:24.925475 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:24.925999 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:24.926014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.926781 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.926784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:24.927641 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:24.927651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:24.928160 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:24.928164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:24.928317 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:24.928792 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:24.928808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:24.930215 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:24.932005 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:24.932011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:24.932025 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:24.932330 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:24.932594 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:24.932598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:24.932599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:24.932771 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:24.933100 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:24.933107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:24.933394 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 03:30:24.933592 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 03:30:24.933933 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260809 03:30:24.935922 23323 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44633
I20260809 03:30:24.936141 23323 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:24.936568 23323 heartbeater.cc:511] Master 127.15.222.126:44633 requested a full tablet report, sending...
I20260809 03:30:24.938231 22980 ts_manager.cc:194] Registered new tserver with Master: 38cb5833d71743ebb893b0f0e297ce4f (127.15.222.65:39303)
I20260809 03:30:24.939837 22980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:52449
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:24.957275 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:25.065261 23332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:25.065497 23332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:25.065558 23332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:25.069409 23332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:25.069464 23332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:25.069542 23332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:25.074536 23332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:25.075470 23332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:25.076656 23332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:25.086019 23338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.086021 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.086019 23344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:25.086174 23332 server_base.cc:1034] running on GCE node
I20260809 03:30:25.086599 23332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:25.087184 23332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:25.088338 23332 hybrid_clock.cc:648] HybridClock initialized: now 1786246225088321 us; error 29 us; skew 500 ppm
Aug 09 03:30:25 dist-test-slave-cvwc krb5kdc[22904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246225, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:25.090904 23332 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:25.091558 23332 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:25.091619 23332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:25.091797 23332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:25.093781 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "94ed88d4cb1d41a9a1c08d843b4bd216"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "39baf7c976f45561f527063ecbdf84c6"
server_key_iv: "107e9d34862c25ac5b691948079eb853"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.094324 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "94ed88d4cb1d41a9a1c08d843b4bd216"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "39baf7c976f45561f527063ecbdf84c6"
server_key_iv: "107e9d34862c25ac5b691948079eb853"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.097792 23332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 03:30:25.100050 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.101147 23332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:25.101248 23332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "94ed88d4cb1d41a9a1c08d843b4bd216"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "39baf7c976f45561f527063ecbdf84c6"
server_key_iv: "107e9d34862c25ac5b691948079eb853"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.101341 23332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:25.123950 23332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:25.126601 23332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:25.126763 23332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:25.127310 23332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:25.128201 23332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:25.128252 23332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.128314 23332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:25.128345 23332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.136552 23332 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:46411
I20260809 03:30:25.136569 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:46411 every 8 connection(s)
I20260809 03:30:25.137072 23332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:25.142849 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 23332
I20260809 03:30:25.142977 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:25.143324 16249 external_mini_cluster.cc:1428] Setting key 1390dde35cde7f4bdf0d2c14e1f5aeec
Aug 09 03:30:25 dist-test-slave-cvwc krb5kdc[22904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246225, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:25.150192 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.138714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:41209 (local address 127.15.222.126:44633)
0809 03:30:25.138847 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.138851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.139948 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 03:30:25.140800 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.140803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.140851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.140923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.142046 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.142573 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.144146 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.144422 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.144515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.146803 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.146807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.146810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:25.146838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.148248 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.148704 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.148707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.148710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.148789 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.149091 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.149094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.149096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.149269 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.149358 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.149913 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.150033 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":50}
I20260809 03:30:25.150909 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.139059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44633 (local address 127.15.222.66:41209)
0809 03:30:25.139812 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.139850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.140586 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.141074 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.141081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.141425 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:25.141907 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.141920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.142711 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.142715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.143991 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.144002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.144573 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.144577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.144779 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.145278 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:25.145294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:25.146654 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:25.148345 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.148349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.148360 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.148604 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.148872 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.148875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.148876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.149002 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.149366 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:25.149373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:25.149611 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.149843 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.150143 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260809 03:30:25.151961 23544 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44633
I20260809 03:30:25.152213 23544 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:25.152719 23544 heartbeater.cc:511] Master 127.15.222.126:44633 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:25.153824 22980 ts_manager.cc:194] Registered new tserver with Master: 94ed88d4cb1d41a9a1c08d843b4bd216 (127.15.222.66:46411)
I20260809 03:30:25.154518 22980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:41209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:25.168094 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:25.277616 23551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:25.277875 23551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:25.277932 23551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:25.281898 23551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:25.281945 23551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:25.282018 23551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:25.287492 23551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:25.288384 23551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:25.289276 23551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:25.298367 23551 server_base.cc:1034] running on GCE node
W20260809 03:30:25.298316 23557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.298328 23560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.298316 23558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:25.298977 23551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:25.299515 23551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:25.300673 23551 hybrid_clock.cc:648] HybridClock initialized: now 1786246225300655 us; error 27 us; skew 500 ppm
Aug 09 03:30:25 dist-test-slave-cvwc krb5kdc[22904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246225, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:25.303227 23551 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:25.303910 23551 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:25.303977 23551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:25.304167 23551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:25.306183 23551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "f71e1667834c434a9a1b2009816a6846"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "12c92b1cf5610c769c6aa6a94296455e"
server_key_iv: "a88421439a331431f1ef9cf2aaf679a2"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.306685 23551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "f71e1667834c434a9a1b2009816a6846"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "12c92b1cf5610c769c6aa6a94296455e"
server_key_iv: "a88421439a331431f1ef9cf2aaf679a2"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.310070 23551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:25.312202 23574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.313295 23551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:25.313395 23551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "f71e1667834c434a9a1b2009816a6846"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "12c92b1cf5610c769c6aa6a94296455e"
server_key_iv: "a88421439a331431f1ef9cf2aaf679a2"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.313481 23551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:25.328176 23551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:25.330749 23551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:25.330921 23551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:25.331465 23551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:25.332283 23551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:25.332334 23551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.332389 23551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:25.332423 23551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.341338 23551 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:44189
I20260809 03:30:25.341359 23746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:44189 every 8 connection(s)
I20260809 03:30:25.341921 23551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:25.343819 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 23551
I20260809 03:30:25.343915 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:25.344172 16249 external_mini_cluster.cc:1428] Setting key 38e30136df4b265cb6408c8368bc6f74
Aug 09 03:30:25 dist-test-slave-cvwc krb5kdc[22904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246225, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:25.354331 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.343561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37603 (local address 127.15.222.126:44633)
0809 03:30:25.343717 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.343722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.344930 (+ 1208us) server_negotiation.cc:407] Connection header received
0809 03:30:25.345883 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.345886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.345940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.346012 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.347132 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.347651 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.348729 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.348957 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.349041 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.351213 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.351216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.351218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:25.351240 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.352482 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.352933 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.352936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.352938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.353010 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.353312 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.353315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.353317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.353486 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.353569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.354058 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.354174 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 03:30:25.355010 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.343874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44633 (local address 127.15.222.67:37603)
0809 03:30:25.344791 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.344843 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.345661 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.346175 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.346182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.346565 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:25.347002 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.347015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.347762 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.347766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.348618 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.348627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.349086 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.349090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.349262 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.349707 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:25.349751 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:25.351069 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:25.352581 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.352587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.352600 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.352841 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.353090 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:25.353093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:25.353095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:25.353220 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:25.353582 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:25.353588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:25.353843 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.354000 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.354294 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260809 03:30:25.355970 23748 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44633
I20260809 03:30:25.356177 23748 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:25.356597 23748 heartbeater.cc:511] Master 127.15.222.126:44633 requested a full tablet report, sending...
I20260809 03:30:25.357636 22980 ts_manager.cc:194] Registered new tserver with Master: f71e1667834c434a9a1b2009816a6846 (127.15.222.67:44189)
I20260809 03:30:25.358060 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:25.358291 22980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:37603
I20260809 03:30:25.366415 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.359923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.15.222.126:44633)
0809 03:30:25.360041 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.360045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.360213 (+ 168us) server_negotiation.cc:407] Connection header received
0809 03:30:25.360302 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.360304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.360339 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.360399 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.360968 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.361437 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.362392 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.362721 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.362809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.363901 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.363903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.363905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:25.363925 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.365133 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.365447 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.365449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.365450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.365515 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:25.365687 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:25.365689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:25.365690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:25.365917 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.366019 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.366173 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.366256 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:25.368851 22980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:25.371709 22980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:25.385149 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.379334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39303 (local address 127.0.0.1:33528)
0809 03:30:25.380031 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.380077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.380201 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.380757 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.380762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.380786 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:25.380925 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.380931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382396 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.382399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.383579 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.383588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384559 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.384679 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.384716 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.384816 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.384976 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":28,"thread_start_us":82,"threads_started":1}
I20260809 03:30:25.385169 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.379517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.15.222.65:39303)
0809 03:30:25.380266 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.380278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.380300 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:30:25.380376 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.380380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.380594 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.380758 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:25.381033 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382273 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.383701 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384434 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.384692 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.384940 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.385033 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":153,"threads_started":1}
I20260809 03:30:25.385201 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.379832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.15.222.67:44189)
0809 03:30:25.380785 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.380800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.380820 (+ 20us) server_negotiation.cc:407] Connection header received
0809 03:30:25.380901 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.380907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.381086 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.381279 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:25.381616 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382481 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.383937 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384664 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384756 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.384871 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.384982 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.385066 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":129,"threads_started":1}
I20260809 03:30:25.385181 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.379686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:44189 (local address 127.0.0.1:50330)
0809 03:30:25.380467 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.380479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.380564 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.381399 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.381402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.381420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:25.381497 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.381503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382616 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.382619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.383816 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.383826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384767 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.384856 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.384878 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.384958 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.385028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":87,"threads_started":1}
I20260809 03:30:25.386054 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.380440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.15.222.66:46411)
0809 03:30:25.381072 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.381083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.381103 (+ 20us) server_negotiation.cc:407] Connection header received
0809 03:30:25.381171 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.381176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.381359 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.381507 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:25.381673 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382677 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.383988 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384511 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384612 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.384748 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.385800 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.385909 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":102,"threads_started":1}
I20260809 03:30:25.387902 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.380033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:46411 (local address 127.0.0.1:44574)
0809 03:30:25.380810 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.380823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.380914 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.381496 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.381499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.381513 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:25.381580 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.381586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.382812 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.382815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.383903 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.383911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.384810 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.384814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.385618 (+ 804us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.385644 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.385731 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.386302 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260809 03:30:25.388180 23651 tablet_service.cc:1467] Processing CreateTablet for tablet 2362612d979b4231963685d3737fe195 (DEFAULT_TABLE table=test-table [id=40ca6b78c36c4b40a06debd1888b3dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:25.388195 23206 tablet_service.cc:1467] Processing CreateTablet for tablet 2362612d979b4231963685d3737fe195 (DEFAULT_TABLE table=test-table [id=40ca6b78c36c4b40a06debd1888b3dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:25.389063 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362612d979b4231963685d3737fe195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:25.389072 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362612d979b4231963685d3737fe195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:25.392344 23441 tablet_service.cc:1467] Processing CreateTablet for tablet 2362612d979b4231963685d3737fe195 (DEFAULT_TABLE table=test-table [id=40ca6b78c36c4b40a06debd1888b3dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:25.393333 23441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362612d979b4231963685d3737fe195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:25.395195 23781 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Bootstrap starting.
I20260809 03:30:25.396459 23782 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Bootstrap starting.
I20260809 03:30:25.397289 23781 tablet_bootstrap.cc:654] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:25.398273 23781 log.cc:824] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:25.398591 23782 tablet_bootstrap.cc:654] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:25.399778 23782 log.cc:824] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:25.400445 23781 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: No bootstrap required, opened a new log
I20260809 03:30:25.400517 23784 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Bootstrap starting.
I20260809 03:30:25.400669 23781 ts_tablet_manager.cc:1397] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:30:25.401556 23782 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: No bootstrap required, opened a new log
I20260809 03:30:25.401767 23782 ts_tablet_manager.cc:1397] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:30:25.402525 23784 tablet_bootstrap.cc:654] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:25.403584 23784 log.cc:824] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:25.404381 23781 raft_consensus.cc:348] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.404608 23781 raft_consensus.cc:374] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:25.404680 23781 raft_consensus.cc:729] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cb5833d71743ebb893b0f0e297ce4f, State: Initialized, Role: FOLLOWER
I20260809 03:30:25.404682 23782 raft_consensus.cc:348] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.405022 23782 raft_consensus.cc:374] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:25.405092 23782 raft_consensus.cc:729] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71e1667834c434a9a1b2009816a6846, State: Initialized, Role: FOLLOWER
I20260809 03:30:25.405117 23781 consensus_queue.cc:260] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.405525 23782 consensus_queue.cc:260] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.405783 23784 tablet_bootstrap.cc:492] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: No bootstrap required, opened a new log
I20260809 03:30:25.405824 23323 heartbeater.cc:503] Master 127.15.222.126:44633 was elected leader, sending a full tablet report...
I20260809 03:30:25.406013 23784 ts_tablet_manager.cc:1397] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:30:25.406086 23781 ts_tablet_manager.cc:1428] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:30:25.406663 23748 heartbeater.cc:503] Master 127.15.222.126:44633 was elected leader, sending a full tablet report...
I20260809 03:30:25.406747 23782 ts_tablet_manager.cc:1428] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:25.410023 23784 raft_consensus.cc:348] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.410279 23784 raft_consensus.cc:374] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:25.410346 23784 raft_consensus.cc:729] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ed88d4cb1d41a9a1c08d843b4bd216, State: Initialized, Role: FOLLOWER
I20260809 03:30:25.410676 23784 consensus_queue.cc:260] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.411217 23544 heartbeater.cc:503] Master 127.15.222.126:44633 was elected leader, sending a full tablet report...
I20260809 03:30:25.411473 23784 ts_tablet_manager.cc:1428] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 03:30:25.422000 23326 tablet.cc:2369] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:25.428972 23792 raft_consensus.cc:482] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:25.429141 23792 raft_consensus.cc:504] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.430219 23792 leader_election.cc:288] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38cb5833d71743ebb893b0f0e297ce4f (127.15.222.65:39303), 94ed88d4cb1d41a9a1c08d843b4bd216 (127.15.222.66:46411)
I20260809 03:30:25.434290 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.430556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:44169 (local address 127.15.222.65:39303)
0809 03:30:25.430685 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.430689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.430703 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:25.430746 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.430748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.430790 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.430859 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:25.431175 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.431870 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.433285 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.433789 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.433890 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.433949 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.434109 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.434176 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:25.434305 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.430427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39303 (local address 127.15.222.67:44169)
0809 03:30:25.430547 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.430560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.430643 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.430899 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.430903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.430926 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:25.431059 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.431068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.431998 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.432001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.433160 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.433172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.433906 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.433909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.433997 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.434019 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.434081 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.434154 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 03:30:25.434669 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.430848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:45015 (local address 127.15.222.66:46411)
0809 03:30:25.430966 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.430970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.431186 (+ 216us) server_negotiation.cc:407] Connection header received
0809 03:30:25.431309 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.431312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.431357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.431447 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:25.431656 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.432226 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.433513 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.434087 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.434193 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.434236 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.434472 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.434552 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:25.435001 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.430713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:46411 (local address 127.15.222.67:45015)
0809 03:30:25.431106 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.431120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.431217 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.431458 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.431460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.431474 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:25.431571 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.431576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.432327 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.432331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.433412 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:25.433420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.434210 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.434214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:25.434307 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.434334 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.434820 (+ 486us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.434899 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":121,"threads_started":1}
I20260809 03:30:25.434983 23246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362612d979b4231963685d3737fe195" candidate_uuid: "f71e1667834c434a9a1b2009816a6846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cb5833d71743ebb893b0f0e297ce4f" is_pre_election: true
I20260809 03:30:25.435256 23246 raft_consensus.cc:2454] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71e1667834c434a9a1b2009816a6846 in term 0.
I20260809 03:30:25.435591 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362612d979b4231963685d3737fe195" candidate_uuid: "f71e1667834c434a9a1b2009816a6846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" is_pre_election: true
I20260809 03:30:25.435719 23575 leader_election.cc:302] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cb5833d71743ebb893b0f0e297ce4f, f71e1667834c434a9a1b2009816a6846; no voters:
I20260809 03:30:25.435817 23478 raft_consensus.cc:2454] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71e1667834c434a9a1b2009816a6846 in term 0.
I20260809 03:30:25.436004 23792 raft_consensus.cc:2781] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:25.436096 23792 raft_consensus.cc:482] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:25.436134 23792 raft_consensus.cc:3037] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:25.437101 23792 raft_consensus.cc:504] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.437397 23792 leader_election.cc:288] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [CANDIDATE]: Term 1 election: Requested vote from peers 38cb5833d71743ebb893b0f0e297ce4f (127.15.222.65:39303), 94ed88d4cb1d41a9a1c08d843b4bd216 (127.15.222.66:46411)
I20260809 03:30:25.437765 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362612d979b4231963685d3737fe195" candidate_uuid: "f71e1667834c434a9a1b2009816a6846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216"
I20260809 03:30:25.437810 23246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362612d979b4231963685d3737fe195" candidate_uuid: "f71e1667834c434a9a1b2009816a6846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cb5833d71743ebb893b0f0e297ce4f"
I20260809 03:30:25.437886 23478 raft_consensus.cc:3037] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:25.437923 23246 raft_consensus.cc:3037] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:25.438856 23478 raft_consensus.cc:2454] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71e1667834c434a9a1b2009816a6846 in term 1.
I20260809 03:30:25.438864 23246 raft_consensus.cc:2454] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71e1667834c434a9a1b2009816a6846 in term 1.
I20260809 03:30:25.439129 23575 leader_election.cc:302] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cb5833d71743ebb893b0f0e297ce4f, f71e1667834c434a9a1b2009816a6846; no voters:
I20260809 03:30:25.439284 23792 raft_consensus.cc:2781] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:25.439481 23792 raft_consensus.cc:686] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 1 LEADER]: Becoming Leader. State: Replica: f71e1667834c434a9a1b2009816a6846, State: Running, Role: LEADER
I20260809 03:30:25.439770 23792 consensus_queue.cc:237] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } }
I20260809 03:30:25.442576 22979 catalog_manager.cc:5495] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 reported cstate change: term changed from 0 to 1, leader changed from <none> to f71e1667834c434a9a1b2009816a6846 (127.15.222.67). New cstate: current_term: 1 leader_uuid: "f71e1667834c434a9a1b2009816a6846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f71e1667834c434a9a1b2009816a6846" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44189 } health_report { overall_health: HEALTHY } } }
I20260809 03:30:25.461444 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.458455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.15.222.67:44189)
0809 03:30:25.458567 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.458570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.458582 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:25.458669 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.458671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.458716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.458793 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:25.458974 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.459476 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.460258 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:25.460455 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:25.460518 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:25.460725 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:25.461124 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:25.461202 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.461286 (+ 84us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.461333 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:25.466748 23246 raft_consensus.cc:1261] T 2362612d979b4231963685d3737fe195 P 38cb5833d71743ebb893b0f0e297ce4f [term 1 FOLLOWER]: Refusing update from remote peer f71e1667834c434a9a1b2009816a6846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:25.466748 23478 raft_consensus.cc:1261] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 1 FOLLOWER]: Refusing update from remote peer f71e1667834c434a9a1b2009816a6846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:25.467355 23792 consensus_queue.cc:1035] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ed88d4cb1d41a9a1c08d843b4bd216" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:25.467541 23795 consensus_queue.cc:1035] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cb5833d71743ebb893b0f0e297ce4f" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:25.474581 23804 mvcc.cc:205] Tried to move back new op lower bound from 7316464539505123328 to 7316464539403673600. Current Snapshot: MvccSnapshot[applied={T|T < 7316464539505123328}]
I20260809 03:30:25.478798 23803 mvcc.cc:205] Tried to move back new op lower bound from 7316464539505123328 to 7316464539403673600. Current Snapshot: MvccSnapshot[applied={T|T < 7316464539505123328 or (T in {7316464539505123328})}]
I20260809 03:30:25.483217 22979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:25.483384 22979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:25.485739 22979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dba87714909846d4822568098234080f: Sending DeleteTablet for 3 replicas of tablet 2362612d979b4231963685d3737fe195
I20260809 03:30:25.486433 23651 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362612d979b4231963685d3737fe195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:50330
I20260809 03:30:25.486475 23206 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362612d979b4231963685d3737fe195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:33528
I20260809 03:30:25.487049 23441 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362612d979b4231963685d3737fe195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:44574
I20260809 03:30:25.487110 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 23084
I20260809 03:30:25.489645 23817 tablet_replica.cc:330] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: stopping tablet replica
I20260809 03:30:25.489938 23817 raft_consensus.cc:2229] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:25.490242 23817 raft_consensus.cc:2258] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:25.493609 23817 ts_tablet_manager.cc:1905] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:25.496183 23821 tablet_replica.cc:330] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: stopping tablet replica
I20260809 03:30:25.496253 23817 ts_tablet_manager.cc:1918] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:25.496373 23817 log.cc:1196] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/2362612d979b4231963685d3737fe195
I20260809 03:30:25.496611 23821 raft_consensus.cc:2229] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:25.496912 23821 raft_consensus.cc:2258] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:25.497383 23817 ts_tablet_manager.cc:1939] T 2362612d979b4231963685d3737fe195 P f71e1667834c434a9a1b2009816a6846: Deleting consensus metadata
I20260809 03:30:25.498392 22957 catalog_manager.cc:4841] TS f71e1667834c434a9a1b2009816a6846 (127.15.222.67:44189): tablet 2362612d979b4231963685d3737fe195 (table test-table [id=40ca6b78c36c4b40a06debd1888b3dd2]) successfully deleted
I20260809 03:30:25.499536 23821 ts_tablet_manager.cc:1905] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:25.501458 23821 ts_tablet_manager.cc:1918] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:25.501541 23821 log.cc:1196] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/2362612d979b4231963685d3737fe195
I20260809 03:30:25.501858 23821 ts_tablet_manager.cc:1939] T 2362612d979b4231963685d3737fe195 P 94ed88d4cb1d41a9a1c08d843b4bd216: Deleting consensus metadata
I20260809 03:30:25.502807 22951 catalog_manager.cc:4841] TS 94ed88d4cb1d41a9a1c08d843b4bd216 (127.15.222.66:46411): tablet 2362612d979b4231963685d3737fe195 (table test-table [id=40ca6b78c36c4b40a06debd1888b3dd2]) successfully deleted
W20260809 03:30:25.506841 22951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:39303 (error 108)
I20260809 03:30:25.506997 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 23332
I20260809 03:30:25.509413 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.509156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39303 (local address 127.0.0.1:33530)
0809 03:30:25.509265 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.509333 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:39303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:30:25.509629 22951 catalog_manager.cc:4584] TS 38cb5833d71743ebb893b0f0e297ce4f (127.15.222.65:39303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2362612d979b4231963685d3737fe195: Network error: Client connection negotiation failed: client connection to 127.15.222.65:39303: connect: Connection refused (error 111)
I20260809 03:30:25.518002 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 23551
I20260809 03:30:25.529102 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 22923
2026-08-09T03:30:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2301 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T03:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:25Z Disabled control of system clock
I20260809 03:30:25.560150 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:39309
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:39309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:30:25.670548 23829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:25.670806 23829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:25.670842 23829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:25.674724 23829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:25.674782 23829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:25.674795 23829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:25.674816 23829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:25.674832 23829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:25.679448 23829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:39309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:25.680449 23829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:25.681839 23829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:25.690065 23835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.690091 23838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.690088 23836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:25.690168 23829 server_base.cc:1034] running on GCE node
I20260809 03:30:25.691509 23829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:25.692556 23829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:25.693732 23829 hybrid_clock.cc:648] HybridClock initialized: now 1786246225693698 us; error 25 us; skew 500 ppm
I20260809 03:30:25.695504 23829 webserver.cc:460] Webserver started at http://127.15.222.126:33997/ using document root <none> and password file <none>
I20260809 03:30:25.696022 23829 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:25.696077 23829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:25.696244 23829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:25.698253 23829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "e7a5c218c8274eaea5c48823f1c74702"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "5a696ad5f934f794e0d58a3e078c4518"
server_key_iv: "8d20aea9e1b06835a93321dc6af28e52"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.698793 23829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "e7a5c218c8274eaea5c48823f1c74702"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "5a696ad5f934f794e0d58a3e078c4518"
server_key_iv: "8d20aea9e1b06835a93321dc6af28e52"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.702279 23829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:25.704535 23850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.705694 23829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:25.705832 23829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e7a5c218c8274eaea5c48823f1c74702"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "5a696ad5f934f794e0d58a3e078c4518"
server_key_iv: "8d20aea9e1b06835a93321dc6af28e52"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.705925 23829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:25.710243 23829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:25.710806 23829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:25.710959 23829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:25.717197 23829 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:39309
I20260809 03:30:25.717242 23938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:39309 every 8 connection(s)
I20260809 03:30:25.718127 23829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:25.721295 23939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:25.725317 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 23829
I20260809 03:30:25.725418 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:25.725651 16249 external_mini_cluster.cc:1428] Setting key 704340ffd31eddbecaffa0142da66f32
I20260809 03:30:25.726914 23939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: Bootstrap starting.
I20260809 03:30:25.729156 23939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:25.729907 23939 log.cc:824] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:25.731158 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.727068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.15.222.126:39309)
0809 03:30:25.727698 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.727713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.727747 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:25.728373 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.728391 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.728726 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.729031 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.729954 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.729959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.729970 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:25.729978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.730027 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:25.730048 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.730150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.730274 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.730513 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260809 03:30:25.731940 23939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: No bootstrap required, opened a new log
I20260809 03:30:25.735525 23939 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } }
I20260809 03:30:25.735730 23939 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:25.735767 23939 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a5c218c8274eaea5c48823f1c74702, State: Initialized, Role: FOLLOWER
I20260809 03:30:25.736133 23939 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } }
I20260809 03:30:25.736253 23939 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:25.736295 23939 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:25.736362 23939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:25.737119 23939 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } }
I20260809 03:30:25.737474 23939 leader_election.cc:302] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a5c218c8274eaea5c48823f1c74702; no voters:
I20260809 03:30:25.737787 23939 leader_election.cc:288] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:25.737859 23948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:25.738040 23948 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [term 1 LEADER]: Becoming Leader. State: Replica: e7a5c218c8274eaea5c48823f1c74702, State: Running, Role: LEADER
I20260809 03:30:25.738325 23948 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } }
I20260809 03:30:25.739025 23939 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:25.740012 23949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a5c218c8274eaea5c48823f1c74702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } } }
I20260809 03:30:25.740047 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a5c218c8274eaea5c48823f1c74702. Latest consensus state: current_term: 1 leader_uuid: "e7a5c218c8274eaea5c48823f1c74702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c218c8274eaea5c48823f1c74702" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 39309 } } }
I20260809 03:30:25.740129 23949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:25.740154 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:25.741138 23960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:25.743687 23960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:25.748673 23960 catalog_manager.cc:1347] Generated new cluster ID: 045e9399082f456e8577b7c09a589419
I20260809 03:30:25.748723 23960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:25.765143 23960 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:25.765511 23960 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:25.775749 23960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: Generated new TSK 0
I20260809 03:30:25.776458 23960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:25.790527 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--builtin_ntp_servers=127.15.222.84:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:30:25.915364 23976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:25.915598 23976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:25.915635 23976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:25.919601 23976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:25.919652 23976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:25.919703 23976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:25.924181 23976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:25.925078 23976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:25.926357 23976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:25.936028 23991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.936048 23989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:25.936442 23995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:25.936582 23976 server_base.cc:1034] running on GCE node
I20260809 03:30:25.937034 23976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:25.937794 23976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:25.938983 23976 hybrid_clock.cc:648] HybridClock initialized: now 1786246225938957 us; error 37 us; skew 500 ppm
I20260809 03:30:25.941160 23976 webserver.cc:460] Webserver started at http://127.15.222.65:37359/ using document root <none> and password file <none>
I20260809 03:30:25.941761 23976 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:25.941833 23976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:25.942037 23976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:25.944000 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "0f839c72b8ce44089283425b7bd44779"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "ec23df001c1343057050b246258d98a8"
server_key_iv: "fb51029cd94f71f1840beb30df68eb7f"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.944545 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "0f839c72b8ce44089283425b7bd44779"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "ec23df001c1343057050b246258d98a8"
server_key_iv: "fb51029cd94f71f1840beb30df68eb7f"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.948040 23976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:25.950369 24005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.951516 23976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:25.951610 23976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "0f839c72b8ce44089283425b7bd44779"
format_stamp: "Formatted at 2026-08-09 03:30:25 on dist-test-slave-cvwc"
server_key: "ec23df001c1343057050b246258d98a8"
server_key_iv: "fb51029cd94f71f1840beb30df68eb7f"
server_key_version: "encryptionkey@0"
I20260809 03:30:25.951705 23976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:25.955989 23976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:25.956539 23976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:25.956686 23976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:25.957177 23976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:25.958104 23976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:25.958150 23976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.958192 23976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:25.958230 23976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:25.966547 23976 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:37605
I20260809 03:30:25.966621 24187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:37605 every 8 connection(s)
I20260809 03:30:25.967386 23976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:25.967722 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 23976
I20260809 03:30:25.967846 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:25.968111 16249 external_mini_cluster.cc:1428] Setting key c609f52a3639692f5a7a986c0fa7b282
I20260809 03:30:25.970319 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--builtin_ntp_servers=127.15.222.84:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 03:30:25.973649 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.969179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47037 (local address 127.15.222.126:39309)
0809 03:30:25.969311 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:25.969316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:25.970425 (+ 1109us) server_negotiation.cc:407] Connection header received
0809 03:30:25.971927 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:25.971929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:25.971965 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:25.972034 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:25.973198 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:25.973202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:25.973204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:25.973215 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:25.973238 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:25.973248 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:25.973333 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:25.973478 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 03:30:25.973521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:25.974509 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:25.969577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:39309 (local address 127.15.222.65:47037)
0809 03:30:25.970282 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:25.970323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:25.971680 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:25.972219 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:25.972237 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:25.972349 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:25.972891 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:25.972893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:25.972951 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:25.973046 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:25.973331 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:25.973339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:25.973347 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 03:30:25.973499 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 03:30:25.973685 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260809 03:30:25.975628 24189 heartbeater.cc:347] Connected to a master server at 127.15.222.126:39309
I20260809 03:30:25.975871 24189 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:25.976339 24189 heartbeater.cc:511] Master 127.15.222.126:39309 requested a full tablet report, sending...
I20260809 03:30:25.977912 23879 ts_manager.cc:194] Registered new tserver with Master: 0f839c72b8ce44089283425b7bd44779 (127.15.222.65:37605)
W20260809 03:30:26.080273 24195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:26.080515 24195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:26.080554 24195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:26.084496 24195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:26.084543 24195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:26.084604 24195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:26.089231 24195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:26.090240 24195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:26.091532 24195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:26.100498 24203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:26.100513 24207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:26.100513 24202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:26.100611 24195 server_base.cc:1034] running on GCE node
I20260809 03:30:26.101125 24195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:26.101706 24195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:26.102892 24195 hybrid_clock.cc:648] HybridClock initialized: now 1786246226102868 us; error 39 us; skew 500 ppm
I20260809 03:30:26.104544 24195 webserver.cc:460] Webserver started at http://127.15.222.66:34207/ using document root <none> and password file <none>
I20260809 03:30:26.105067 24195 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:26.105131 24195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:26.105325 24195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:26.107327 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "ca7e0d7719124570919dc81faa0dadf0"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "faf9f5e40614469e2525bcea003f976c"
server_key_iv: "6e10b3cd93f1e72124d0a86ab78b821b"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.107857 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "ca7e0d7719124570919dc81faa0dadf0"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "faf9f5e40614469e2525bcea003f976c"
server_key_iv: "6e10b3cd93f1e72124d0a86ab78b821b"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.111450 24195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:30:26.113819 24216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.114964 24195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:26.115065 24195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "ca7e0d7719124570919dc81faa0dadf0"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "faf9f5e40614469e2525bcea003f976c"
server_key_iv: "6e10b3cd93f1e72124d0a86ab78b821b"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.115159 24195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:26.119470 24195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:26.120018 24195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:26.120163 24195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:26.120680 24195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:26.121536 24195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:26.121590 24195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.121639 24195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:26.121665 24195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.129545 24195 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:43991
I20260809 03:30:26.129575 24407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:43991 every 8 connection(s)
I20260809 03:30:26.130352 24195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:26.135666 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.131995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46303 (local address 127.15.222.126:39309)
0809 03:30:26.132130 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.132133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.133049 (+ 916us) server_negotiation.cc:407] Connection header received
0809 03:30:26.134317 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.134319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.134357 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.134422 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.135321 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.135324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.135326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.135337 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.135355 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.135363 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.135439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.135525 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.135561 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:26.136422 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.132315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:39309 (local address 127.15.222.66:46303)
0809 03:30:26.132927 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.132966 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.134129 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.134561 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.134567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.134647 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.135096 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.135098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.135134 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.135197 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.135432 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.135437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.135440 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.135556 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.135731 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260809 03:30:26.137307 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 24195
I20260809 03:30:26.137422 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:26.137513 24408 heartbeater.cc:347] Connected to a master server at 127.15.222.126:39309
I20260809 03:30:26.137696 16249 external_mini_cluster.cc:1428] Setting key d0d3dfce2c3e6cb40f0f96c02a15bd46
I20260809 03:30:26.137734 24408 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:26.138108 24408 heartbeater.cc:511] Master 127.15.222.126:39309 requested a full tablet report, sending...
I20260809 03:30:26.139099 23879 ts_manager.cc:194] Registered new tserver with Master: ca7e0d7719124570919dc81faa0dadf0 (127.15.222.66:43991)
I20260809 03:30:26.139663 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--builtin_ntp_servers=127.15.222.84:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 03:30:26.247962 24411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:26.248186 24411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:26.248240 24411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:26.252137 24411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:26.252184 24411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:26.252255 24411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:26.257193 24411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:39309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:26.258172 24411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:26.259519 24411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:26.268316 24422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:26.268404 24421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:26.268450 24411 server_base.cc:1034] running on GCE node
W20260809 03:30:26.268635 24427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:26.269014 24411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:26.269572 24411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:26.270718 24411 hybrid_clock.cc:648] HybridClock initialized: now 1786246226270719 us; error 44 us; skew 500 ppm
I20260809 03:30:26.272349 24411 webserver.cc:460] Webserver started at http://127.15.222.67:43401/ using document root <none> and password file <none>
I20260809 03:30:26.272879 24411 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:26.272941 24411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:26.273136 24411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:26.275163 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "9221e4b7f396429aaf033c3ab5a41e0a"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "6d5a147bb5c2e9205eff1638f558e4e7"
server_key_iv: "088f9e487f0198af8d5184c4a12747c1"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.275688 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "9221e4b7f396429aaf033c3ab5a41e0a"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "6d5a147bb5c2e9205eff1638f558e4e7"
server_key_iv: "088f9e487f0198af8d5184c4a12747c1"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.279248 24411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 03:30:26.281519 24434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.282668 24411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:26.282770 24411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "9221e4b7f396429aaf033c3ab5a41e0a"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "6d5a147bb5c2e9205eff1638f558e4e7"
server_key_iv: "088f9e487f0198af8d5184c4a12747c1"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.282863 24411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:26.287132 24411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:26.287684 24411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:26.287828 24411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:26.288313 24411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:26.289149 24411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:26.289196 24411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.289243 24411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:26.289269 24411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.297435 24411 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:43711
I20260809 03:30:26.297464 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:43711 every 8 connection(s)
I20260809 03:30:26.298285 24411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:26.303627 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.299908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:42267 (local address 127.15.222.126:39309)
0809 03:30:26.300066 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.300070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.301016 (+ 946us) server_negotiation.cc:407] Connection header received
0809 03:30:26.302300 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.302303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.302340 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.302415 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.303286 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.303289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.303291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.303301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.303321 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.303328 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.303403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.303489 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.303525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:30:26.304373 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.300285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:39309 (local address 127.15.222.67:42267)
0809 03:30:26.300877 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.300919 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.302129 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.302540 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.302547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.302629 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.303065 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.303066 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.303103 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.303165 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.303388 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.303401 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.303404 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.303518 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.303704 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260809 03:30:26.305344 24628 heartbeater.cc:347] Connected to a master server at 127.15.222.126:39309
I20260809 03:30:26.305576 24628 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:26.305730 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 24411
I20260809 03:30:26.305836 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:26.306063 24628 heartbeater.cc:511] Master 127.15.222.126:39309 requested a full tablet report, sending...
I20260809 03:30:26.306146 16249 external_mini_cluster.cc:1428] Setting key 47703e519fe8c30a74d53c12df72cecd
I20260809 03:30:26.307024 23878 ts_manager.cc:194] Registered new tserver with Master: 9221e4b7f396429aaf033c3ab5a41e0a (127.15.222.67:43711)
I20260809 03:30:26.308075 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:26.311280 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.309839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.15.222.126:39309)
0809 03:30:26.309946 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.309950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.310141 (+ 191us) server_negotiation.cc:407] Connection header received
0809 03:30:26.310244 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.310248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.310280 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.310338 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.310954 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.310958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.310960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.310971 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.310990 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.310997 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.311061 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.311135 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.311171 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:26.313769 23879 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:26.316498 23879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:26.327040 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:43991 (local address 127.0.0.1:38548)
0809 03:30:26.324931 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.324943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.324998 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.325671 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.325674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.325743 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.326267 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.326269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.326288 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.326354 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.326720 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.326729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.326734 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.326840 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.326908 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":91,"threads_started":1}
I20260809 03:30:26.327164 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.15.222.67:43711)
0809 03:30:26.325210 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.325221 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.325244 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:30:26.325294 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.325299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.325472 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.325602 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.326557 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.326563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.326574 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.326586 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.326626 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.326654 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.326738 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.326976 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.327037 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":151,"threads_started":1}
I20260809 03:30:26.327095 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.15.222.66:43991)
0809 03:30:26.325252 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.325263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.325287 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:30:26.325341 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.325347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.325521 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.326007 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.326544 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.326549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.326559 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.326568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.326605 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.326628 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.326734 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.326864 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.326945 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":116,"threads_started":1}
I20260809 03:30:26.327423 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43711 (local address 127.0.0.1:53068)
0809 03:30:26.324509 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.324553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.324643 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.325850 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.325853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.325866 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.326383 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.326386 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.326397 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.326425 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.326775 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.326778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.326781 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.326840 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.326915 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260809 03:30:26.327642 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37605 (local address 127.0.0.1:42064)
0809 03:30:26.324851 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.324864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.324937 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.325638 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.325644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.325739 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.326513 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.326515 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.326523 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.326546 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.327455 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.327458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.327462 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.327531 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.327563 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":89,"threads_started":1}
I20260809 03:30:26.327773 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.324949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.15.222.65:37605)
0809 03:30:26.325245 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.325256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.325274 (+ 18us) server_negotiation.cc:407] Connection header received
0809 03:30:26.325317 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.325322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.325472 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.326493 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.327049 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.327054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.327062 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.327072 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.327193 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.327276 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.327528 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.327574 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.327636 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260809 03:30:26.330165 24523 tablet_service.cc:1467] Processing CreateTablet for tablet d89220ad95a14b0aa6c0a408af132e5d (DEFAULT_TABLE table=test-table [id=03cade6835b140b99b0759dfed700b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:26.330243 24081 tablet_service.cc:1467] Processing CreateTablet for tablet d89220ad95a14b0aa6c0a408af132e5d (DEFAULT_TABLE table=test-table [id=03cade6835b140b99b0759dfed700b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:26.330181 24286 tablet_service.cc:1467] Processing CreateTablet for tablet d89220ad95a14b0aa6c0a408af132e5d (DEFAULT_TABLE table=test-table [id=03cade6835b140b99b0759dfed700b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:26.331084 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89220ad95a14b0aa6c0a408af132e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:26.331174 24286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89220ad95a14b0aa6c0a408af132e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:26.331174 24081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d89220ad95a14b0aa6c0a408af132e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:26.337062 24654 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Bootstrap starting.
I20260809 03:30:26.338546 24655 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Bootstrap starting.
I20260809 03:30:26.339222 24654 tablet_bootstrap.cc:654] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:26.340004 24654 log.cc:824] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:26.340309 24655 tablet_bootstrap.cc:654] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:26.340701 24656 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Bootstrap starting.
I20260809 03:30:26.341053 24655 log.cc:824] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:26.342648 24656 tablet_bootstrap.cc:654] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:26.342686 24655 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: No bootstrap required, opened a new log
I20260809 03:30:26.342854 24655 ts_tablet_manager.cc:1397] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 03:30:26.342955 24654 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: No bootstrap required, opened a new log
I20260809 03:30:26.343132 24654 ts_tablet_manager.cc:1397] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:30:26.343418 24656 log.cc:824] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:26.345224 24656 tablet_bootstrap.cc:492] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: No bootstrap required, opened a new log
I20260809 03:30:26.345412 24656 ts_tablet_manager.cc:1397] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:30:26.346491 24655 raft_consensus.cc:348] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.346691 24655 raft_consensus.cc:374] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:26.346740 24655 raft_consensus.cc:729] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7e0d7719124570919dc81faa0dadf0, State: Initialized, Role: FOLLOWER
I20260809 03:30:26.347126 24655 consensus_queue.cc:260] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.347218 24654 raft_consensus.cc:348] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.347460 24654 raft_consensus.cc:374] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:26.347515 24654 raft_consensus.cc:729] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9221e4b7f396429aaf033c3ab5a41e0a, State: Initialized, Role: FOLLOWER
I20260809 03:30:26.347826 24408 heartbeater.cc:503] Master 127.15.222.126:39309 was elected leader, sending a full tablet report...
I20260809 03:30:26.347934 24654 consensus_queue.cc:260] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.348089 24655 ts_tablet_manager.cc:1428] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:30:26.348510 24628 heartbeater.cc:503] Master 127.15.222.126:39309 was elected leader, sending a full tablet report...
I20260809 03:30:26.348816 24654 ts_tablet_manager.cc:1428] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260809 03:30:26.350615 24656 raft_consensus.cc:348] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.350858 24656 raft_consensus.cc:374] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:26.350916 24656 raft_consensus.cc:729] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f839c72b8ce44089283425b7bd44779, State: Initialized, Role: FOLLOWER
I20260809 03:30:26.351341 24656 consensus_queue.cc:260] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.351965 24189 heartbeater.cc:503] Master 127.15.222.126:39309 was elected leader, sending a full tablet report...
I20260809 03:30:26.352214 24656 ts_tablet_manager.cc:1428] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260809 03:30:26.381253 24409 tablet.cc:2369] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:26.434765 24666 raft_consensus.cc:482] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:26.434973 24666 raft_consensus.cc:504] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.436096 24666 leader_election.cc:288] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9221e4b7f396429aaf033c3ab5a41e0a (127.15.222.67:43711), ca7e0d7719124570919dc81faa0dadf0 (127.15.222.66:43991)
I20260809 03:30:26.437912 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.436532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:55641 (local address 127.15.222.66:43991)
0809 03:30:26.436663 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.436667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.436681 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:26.436727 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.436730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.436768 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.436837 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.437499 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.437503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.437506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.437521 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.437544 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.437555 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.437638 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.437705 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.437772 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:26.437927 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.436496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:34955 (local address 127.15.222.67:43711)
0809 03:30:26.436623 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.436628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.436896 (+ 268us) server_negotiation.cc:407] Connection header received
0809 03:30:26.436962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.436964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.436995 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.437053 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.437568 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.437571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.437573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.437585 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.437603 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.437612 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.437689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.437794 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.437828 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:30:26.437914 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.436334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:43991 (local address 127.15.222.65:55641)
0809 03:30:26.436476 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.436492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.436581 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.436896 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.436899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.436912 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.437334 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.437336 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.437345 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.437375 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.437634 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.437637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.437639 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.437708 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.437782 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 03:30:26.437928 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.436333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43711 (local address 127.15.222.65:34955)
0809 03:30:26.436797 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.436807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.436892 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.437094 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.437097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.437111 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:26.437428 (+ 317us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:26.437429 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:26.437437 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:26.437458 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:26.437681 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:26.437684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:26.437687 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:26.437788 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 03:30:26.437812 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":129,"thread_start_us":142,"threads_started":1}
I20260809 03:30:26.438650 24555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89220ad95a14b0aa6c0a408af132e5d" candidate_uuid: "0f839c72b8ce44089283425b7bd44779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" is_pre_election: true
I20260809 03:30:26.438655 24330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89220ad95a14b0aa6c0a408af132e5d" candidate_uuid: "0f839c72b8ce44089283425b7bd44779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7e0d7719124570919dc81faa0dadf0" is_pre_election: true
I20260809 03:30:26.438913 24330 raft_consensus.cc:2454] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f839c72b8ce44089283425b7bd44779 in term 0.
I20260809 03:30:26.438931 24555 raft_consensus.cc:2454] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f839c72b8ce44089283425b7bd44779 in term 0.
I20260809 03:30:26.439350 24008 leader_election.cc:302] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f839c72b8ce44089283425b7bd44779, 9221e4b7f396429aaf033c3ab5a41e0a; no voters:
I20260809 03:30:26.439591 24666 raft_consensus.cc:2781] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:26.439669 24666 raft_consensus.cc:482] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:26.439716 24666 raft_consensus.cc:3037] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:26.440779 24666 raft_consensus.cc:504] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.441052 24666 leader_election.cc:288] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [CANDIDATE]: Term 1 election: Requested vote from peers 9221e4b7f396429aaf033c3ab5a41e0a (127.15.222.67:43711), ca7e0d7719124570919dc81faa0dadf0 (127.15.222.66:43991)
I20260809 03:30:26.441371 24555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89220ad95a14b0aa6c0a408af132e5d" candidate_uuid: "0f839c72b8ce44089283425b7bd44779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9221e4b7f396429aaf033c3ab5a41e0a"
I20260809 03:30:26.441421 24330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d89220ad95a14b0aa6c0a408af132e5d" candidate_uuid: "0f839c72b8ce44089283425b7bd44779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7e0d7719124570919dc81faa0dadf0"
I20260809 03:30:26.441514 24330 raft_consensus.cc:3037] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:26.441511 24555 raft_consensus.cc:3037] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:26.442633 24330 raft_consensus.cc:2454] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f839c72b8ce44089283425b7bd44779 in term 1.
I20260809 03:30:26.442648 24555 raft_consensus.cc:2454] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f839c72b8ce44089283425b7bd44779 in term 1.
I20260809 03:30:26.442934 24013 leader_election.cc:302] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f839c72b8ce44089283425b7bd44779, ca7e0d7719124570919dc81faa0dadf0; no voters:
I20260809 03:30:26.443087 24666 raft_consensus.cc:2781] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:26.443323 24666 raft_consensus.cc:686] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [term 1 LEADER]: Becoming Leader. State: Replica: 0f839c72b8ce44089283425b7bd44779, State: Running, Role: LEADER
I20260809 03:30:26.443660 24666 consensus_queue.cc:237] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } }
I20260809 03:30:26.446583 23878 catalog_manager.cc:5495] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f839c72b8ce44089283425b7bd44779 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "0f839c72b8ce44089283425b7bd44779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f839c72b8ce44089283425b7bd44779" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 } health_report { overall_health: UNKNOWN } } }
W20260809 03:30:26.469228 24193 tablet.cc:2369] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:26.477280 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.475878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.15.222.65:37605)
0809 03:30:26.476017 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.476021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.476035 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:26.476155 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.476157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.476216 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.476288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.476927 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.476930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.476933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.476945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.476966 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.476973 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.477049 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.477116 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.477156 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:26.482861 24555 raft_consensus.cc:1261] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 1 FOLLOWER]: Refusing update from remote peer 0f839c72b8ce44089283425b7bd44779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:26.482861 24330 raft_consensus.cc:1261] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 1 FOLLOWER]: Refusing update from remote peer 0f839c72b8ce44089283425b7bd44779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:26.483603 24666 consensus_queue.cc:1035] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9221e4b7f396429aaf033c3ab5a41e0a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:26.483788 24670 consensus_queue.cc:1035] T d89220ad95a14b0aa6c0a408af132e5d P 0f839c72b8ce44089283425b7bd44779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7e0d7719124570919dc81faa0dadf0" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:26.489876 24673 mvcc.cc:205] Tried to move back new op lower bound from 7316464543666606080 to 7316464543515504640. Current Snapshot: MvccSnapshot[applied={T|T < 7316464543666606080}]
I20260809 03:30:26.490970 24675 mvcc.cc:205] Tried to move back new op lower bound from 7316464543666606080 to 7316464543515504640. Current Snapshot: MvccSnapshot[applied={T|T < 7316464543666606080}]
I20260809 03:30:26.499257 23878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:26.499413 23878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:26.501506 23878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7a5c218c8274eaea5c48823f1c74702: Sending DeleteTablet for 3 replicas of tablet d89220ad95a14b0aa6c0a408af132e5d
I20260809 03:30:26.502197 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet d89220ad95a14b0aa6c0a408af132e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:26 UTC) from {username='slave'} at 127.0.0.1:53068
I20260809 03:30:26.502194 24081 tablet_service.cc:1514] Processing DeleteTablet for tablet d89220ad95a14b0aa6c0a408af132e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:26 UTC) from {username='slave'} at 127.0.0.1:42064
I20260809 03:30:26.502328 24286 tablet_service.cc:1514] Processing DeleteTablet for tablet d89220ad95a14b0aa6c0a408af132e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:26 UTC) from {username='slave'} at 127.0.0.1:38548
I20260809 03:30:26.502714 24691 tablet_replica.cc:330] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: stopping tablet replica
I20260809 03:30:26.502877 24688 tablet_replica.cc:330] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: stopping tablet replica
I20260809 03:30:26.503158 24688 raft_consensus.cc:2229] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:26.503351 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 23976
I20260809 03:30:26.503517 24688 raft_consensus.cc:2258] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:26.509110 24691 raft_consensus.cc:2229] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:26.509413 24691 raft_consensus.cc:2258] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:26.509790 24688 ts_tablet_manager.cc:1905] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:26.510943 24691 ts_tablet_manager.cc:1905] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:26.512526 24688 ts_tablet_manager.cc:1918] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:26.512626 24688 log.cc:1196] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/d89220ad95a14b0aa6c0a408af132e5d
I20260809 03:30:26.512965 24688 ts_tablet_manager.cc:1939] T d89220ad95a14b0aa6c0a408af132e5d P 9221e4b7f396429aaf033c3ab5a41e0a: Deleting consensus metadata
I20260809 03:30:26.513537 24691 ts_tablet_manager.cc:1918] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:26.513633 24691 log.cc:1196] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/d89220ad95a14b0aa6c0a408af132e5d
I20260809 03:30:26.513953 24691 ts_tablet_manager.cc:1939] T d89220ad95a14b0aa6c0a408af132e5d P ca7e0d7719124570919dc81faa0dadf0: Deleting consensus metadata
I20260809 03:30:26.514284 23853 catalog_manager.cc:4841] TS 9221e4b7f396429aaf033c3ab5a41e0a (127.15.222.67:43711): tablet d89220ad95a14b0aa6c0a408af132e5d (table test-table [id=03cade6835b140b99b0759dfed700b0a]) successfully deleted
I20260809 03:30:26.514919 23854 catalog_manager.cc:4841] TS ca7e0d7719124570919dc81faa0dadf0 (127.15.222.66:43991): tablet d89220ad95a14b0aa6c0a408af132e5d (table test-table [id=03cade6835b140b99b0759dfed700b0a]) successfully deleted
W20260809 03:30:26.515406 23858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:37605 (error 108)
I20260809 03:30:26.515636 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 24195
I20260809 03:30:26.518317 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.518029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37605 (local address 127.0.0.1:42082)
0809 03:30:26.518182 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.518243 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260809 03:30:26.518500 23858 catalog_manager.cc:4584] TS 0f839c72b8ce44089283425b7bd44779 (127.15.222.65:37605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d89220ad95a14b0aa6c0a408af132e5d: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37605: connect: Connection refused (error 111)
I20260809 03:30:26.531278 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 24411
I20260809 03:30:26.542111 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 23829
2026-08-09T03:30:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1014 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 03:30:26.566689 16249 mini_oidc.cc:152] Starting JWKS server
I20260809 03:30:26.568213 16249 webserver.cc:460] Webserver started at https://127.0.0.1:35829/ using document root <none> and password file <none>
I20260809 03:30:26.568352 16249 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:30:26.568570 16249 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
2026-08-09T03:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:26Z Disabled control of system clock
I20260809 03:30:26.577565 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38551
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:38217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38551
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/testchainca.cert with env {}
W20260809 03:30:26.685003 24704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:26.685266 24704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:26.685319 24704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:26.689198 24704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:26.689248 24704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:26.689268 24704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:26.689287 24704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:26.689307 24704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:30:26.689337 24704 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:30:26.689363 24704 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35829/jwks.json
I20260809 03:30:26.694494 24704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:26.695473 24704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:26.696434 24704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:26.704638 24711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:26.704763 24704 server_base.cc:1034] running on GCE node
W20260809 03:30:26.704638 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:26.704667 24716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:26.705507 24704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:26.706545 24704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:26.707690 24704 hybrid_clock.cc:648] HybridClock initialized: now 1786246226707667 us; error 35 us; skew 500 ppm
I20260809 03:30:26.709372 24704 webserver.cc:460] Webserver started at http://127.15.222.126:45339/ using document root <none> and password file <none>
I20260809 03:30:26.709970 24704 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:26.710034 24704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:26.710227 24704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:26.712260 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "acd7911287904f1cbdd573720b6f0ca2"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "cf85f7974c822c504b5e800171458854"
server_key_iv: "595572099a4e93d59d0e7a639389c703"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.712849 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "acd7911287904f1cbdd573720b6f0ca2"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "cf85f7974c822c504b5e800171458854"
server_key_iv: "595572099a4e93d59d0e7a639389c703"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.716420 24704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:30:26.718621 24731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:26.719820 24704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:26.719928 24704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "acd7911287904f1cbdd573720b6f0ca2"
format_stamp: "Formatted at 2026-08-09 03:30:26 on dist-test-slave-cvwc"
server_key: "cf85f7974c822c504b5e800171458854"
server_key_iv: "595572099a4e93d59d0e7a639389c703"
server_key_version: "encryptionkey@0"
I20260809 03:30:26.720016 24704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:26.732726 24704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:26.733282 24704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:26.733449 24704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:26.739667 24704 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38551
I20260809 03:30:26.739703 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38551 every 8 connection(s)
I20260809 03:30:26.740509 24704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:26.742213 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 24704
I20260809 03:30:26.742348 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:26.742650 16249 external_mini_cluster.cc:1428] Setting key e5afddbd66a8067a6174aa2b5b6fa27e
I20260809 03:30:26.744045 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:26.749408 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2: Bootstrap starting.
I20260809 03:30:26.751001 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.743879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.15.222.126:38551)
0809 03:30:26.744481 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.744496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.744531 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:30:26.745138 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.745153 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.745518 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.745826 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:26.746220 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.747134 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.747975 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.748440 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.748583 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:26.749394 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:26.749399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:26.749411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:26.749829 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:26.749881 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:26.749903 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:26.749987 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:26.750117 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 03:30:26.750349 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260809 03:30:26.751745 24813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:26.752501 24813 log.cc:824] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:26.753782 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:26.754575 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2: No bootstrap required, opened a new log
I20260809 03:30:26.757998 24813 raft_consensus.cc:348] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } }
I20260809 03:30:26.758205 24813 raft_consensus.cc:374] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:26.758255 24813 raft_consensus.cc:729] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd7911287904f1cbdd573720b6f0ca2, State: Initialized, Role: FOLLOWER
I20260809 03:30:26.758647 24813 consensus_queue.cc:260] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } }
I20260809 03:30:26.758764 24813 raft_consensus.cc:388] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:26.758824 24813 raft_consensus.cc:482] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:26.758922 24813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:26.759747 24813 raft_consensus.cc:504] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } }
I20260809 03:30:26.760089 24813 leader_election.cc:302] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acd7911287904f1cbdd573720b6f0ca2; no voters:
I20260809 03:30:26.760398 24813 leader_election.cc:288] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:26.760481 24819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:26.760690 24819 raft_consensus.cc:686] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [term 1 LEADER]: Becoming Leader. State: Replica: acd7911287904f1cbdd573720b6f0ca2, State: Running, Role: LEADER
I20260809 03:30:26.760989 24819 consensus_queue.cc:237] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } }
I20260809 03:30:26.761523 24813 sys_catalog.cc:564] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:26.762517 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acd7911287904f1cbdd573720b6f0ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } } }
I20260809 03:30:26.762650 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:26.763011 24833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:26.762979 24823 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd7911287904f1cbdd573720b6f0ca2. Latest consensus state: current_term: 1 leader_uuid: "acd7911287904f1cbdd573720b6f0ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd7911287904f1cbdd573720b6f0ca2" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38551 } } }
I20260809 03:30:26.765807 24823 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:26.766207 24833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:26.772097 24833 catalog_manager.cc:1347] Generated new cluster ID: 4ec35337acb24c168e62f53df346e3e1
I20260809 03:30:26.772169 24833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:26.797828 24833 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:26.798794 24833 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:26.805399 24833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acd7911287904f1cbdd573720b6f0ca2: Generated new TSK 0
I20260809 03:30:26.806105 24833 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 03:30:26.842605 24866 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 03:30:26.842769 24866 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 03:30:26.843359 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:26.824881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.15.222.126:38551)
0809 03:30:26.825012 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:26.825016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:26.825215 (+ 199us) server_negotiation.cc:407] Connection header received
0809 03:30:26.825433 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:26.825436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:26.825484 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:26.825563 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:30:26.825830 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.826357 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.827545 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.827746 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.827812 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:26.828145 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:30:26.842941 (+ 14796us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 03:30:26.843207 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 03:30:26.843449 24817 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 03:30:26.844520 24865 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:26.824818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38551 (local address 127.0.0.1:40174)
0809 03:30:26.825134 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:26.825150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:26.825291 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:26.825587 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:26.825590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:26.825598 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:30:26.825746 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:26.825752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.826461 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.826465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:26.827435 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:26.827448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:26.827844 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:26.827848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:26.827938 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:26.827990 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:30:26.843144 (+ 15154us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 03:30:26.843381 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.222.126:38551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.222.126:38551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 03:30:26.849215 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 24704
2026-08-09T03:30:26Z chronyd exiting
I20260809 03:30:26.894260 16249 test_util.cc:175] -----------------------------------------------
I20260809 03:30:26.894340 16249 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786246200571663-16249-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 03:30:26.895560 16249 mini_oidc.cc:152] Starting JWKS server
I20260809 03:30:26.896365 16249 webserver.cc:460] Webserver started at https://127.0.0.1:36831/ using document root <none> and password file <none>
I20260809 03:30:26.896464 16249 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:30:26.896720 16249 webserver.cc:460] Webserver started at http://127.0.0.1:42415/ using document root <none> and password file <none>
2026-08-09T03:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:26Z Disabled control of system clock
I20260809 03:30:26.905797 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38139
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38139
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36831/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/testchainca.cert with env {}
W20260809 03:30:27.013377 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:27.013625 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:27.013661 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:27.017537 24878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:27.017587 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:27.017601 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:27.017616 24878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:27.017637 24878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:30:27.017652 24878 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:30:27.017668 24878 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36831/jwks.json
I20260809 03:30:27.022724 24878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36831/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:27.023608 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:27.024520 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:27.032367 24892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:27.032364 24896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:27.032377 24890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:27.032660 24878 server_base.cc:1034] running on GCE node
I20260809 03:30:27.033229 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:27.034263 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:27.035432 24878 hybrid_clock.cc:648] HybridClock initialized: now 1786246227035414 us; error 29 us; skew 500 ppm
I20260809 03:30:27.037240 24878 webserver.cc:460] Webserver started at http://127.15.222.126:37391/ using document root <none> and password file <none>
I20260809 03:30:27.037804 24878 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:27.037876 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:27.038071 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:27.040109 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "9c407c65660d494ca15a1678d0ba183c"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "a7027fccb036db101bd91288bf86faa6"
server_key_iv: "101cbfc328138ea9e82fc50468251dad"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.040625 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "9c407c65660d494ca15a1678d0ba183c"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "a7027fccb036db101bd91288bf86faa6"
server_key_iv: "101cbfc328138ea9e82fc50468251dad"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.044191 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:27.046388 24903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:27.047478 24878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:27.047588 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "9c407c65660d494ca15a1678d0ba183c"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "a7027fccb036db101bd91288bf86faa6"
server_key_iv: "101cbfc328138ea9e82fc50468251dad"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.047676 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:27.073352 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:27.074074 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:27.074245 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:27.080840 24878 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38139
I20260809 03:30:27.080863 24998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38139 every 8 connection(s)
I20260809 03:30:27.081786 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:27.084828 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:27.090060 24999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: Bootstrap starting.
I20260809 03:30:27.090628 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 24878
I20260809 03:30:27.090737 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:27.090989 16249 external_mini_cluster.cc:1428] Setting key 8d2855e69a1cf13a31f338a295acd08c
I20260809 03:30:27.092626 24999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:27.093487 24999 log.cc:824] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:27.095623 24999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: No bootstrap required, opened a new log
I20260809 03:30:27.099790 24999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } }
I20260809 03:30:27.100059 24999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:27.100136 24999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c407c65660d494ca15a1678d0ba183c, State: Initialized, Role: FOLLOWER
I20260809 03:30:27.100615 24999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } }
I20260809 03:30:27.100768 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.092189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.15.222.126:38139)
0809 03:30:27.092864 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.092884 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.092930 (+ 46us) server_negotiation.cc:407] Connection header received
0809 03:30:27.093736 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.093761 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.094140 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.094475 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:27.094879 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.096243 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.097125 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.097858 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.098008 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.098713 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:27.098719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:27.098736 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:27.099236 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:27.099304 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:27.099329 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:27.099432 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:27.099539 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 03:30:27.099801 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260809 03:30:27.100775 24999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:27.100893 24999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:27.100987 24999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:27.101848 24999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } }
I20260809 03:30:27.102208 24999 leader_election.cc:302] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c407c65660d494ca15a1678d0ba183c; no voters:
I20260809 03:30:27.102515 24999 leader_election.cc:288] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:27.102608 25008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:27.102804 25008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [term 1 LEADER]: Becoming Leader. State: Replica: 9c407c65660d494ca15a1678d0ba183c, State: Running, Role: LEADER
I20260809 03:30:27.103125 25008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } }
I20260809 03:30:27.103543 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:27.103722 24999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:27.104239 25008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c407c65660d494ca15a1678d0ba183c. Latest consensus state: current_term: 1 leader_uuid: "9c407c65660d494ca15a1678d0ba183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } } }
I20260809 03:30:27.104349 25008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:27.104622 25010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c407c65660d494ca15a1678d0ba183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c407c65660d494ca15a1678d0ba183c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38139 } } }
I20260809 03:30:27.104713 25010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c [sys.catalog]: This master's current role is: LEADER
W20260809 03:30:27.108091 25030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:27.108156 25030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:27.108220 25016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:27.108903 25016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:27.114095 25016 catalog_manager.cc:1347] Generated new cluster ID: 9705e5a5d59847e589f21499acbee4b8
I20260809 03:30:27.114148 25016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:27.129952 25016 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:27.130859 25016 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:27.137997 25016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c407c65660d494ca15a1678d0ba183c: Generated new TSK 0
I20260809 03:30:27.138586 25016 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 03:30:27.167971 25041 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 03:30:27.168037 25041 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 03:30:27.168725 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.153403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.15.222.126:38139)
0809 03:30:27.153522 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.153527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.153665 (+ 138us) server_negotiation.cc:407] Connection header received
0809 03:30:27.153821 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.153824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.153882 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.153991 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:30:27.154170 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.154998 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.155774 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.156115 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.156212 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.156438 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:30:27.168343 (+ 11905us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 03:30:27.168593 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 03:30:27.168763 25040 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:27.153291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38139 (local address 127.0.0.1:40080)
0809 03:30:27.153583 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.153595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:27.153687 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:27.153982 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:27.153984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:27.153990 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:30:27.154082 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.154086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.155082 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.155084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.155677 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.155684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.156208 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.156211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.156268 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.156288 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:30:27.168510 (+ 12222us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 03:30:27.168635 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.222.126:38139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
W20260809 03:30:27.168787 25005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 03:30:27.169148 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 24878
2026-08-09T03:30:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 03:30:27.215921 16249 mini_oidc.cc:152] Starting JWKS server
I20260809 03:30:27.216744 16249 webserver.cc:460] Webserver started at https://127.0.0.1:36651/ using document root <none> and password file <none>
I20260809 03:30:27.216838 16249 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 03:30:27.217082 16249 webserver.cc:460] Webserver started at http://127.0.0.1:44051/ using document root <none> and password file <none>
2026-08-09T03:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:27Z Disabled control of system clock
I20260809 03:30:27.226171 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41045
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41045
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 03:30:27.333449 25055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:27.333691 25055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:27.333760 25055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:27.337636 25055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:27.337685 25055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:27.337699 25055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:27.337747 25055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:27.337776 25055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:30:27.337806 25055 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:30:27.337832 25055 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36651/jwks.json
I20260809 03:30:27.342397 25055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:27.343334 25055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:27.344614 25055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:27.352532 25055 server_base.cc:1034] running on GCE node
W20260809 03:30:27.352492 25061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:27.352487 25062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:27.352497 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:27.353255 25055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:27.354288 25055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:27.355432 25055 hybrid_clock.cc:648] HybridClock initialized: now 1786246227355412 us; error 32 us; skew 500 ppm
I20260809 03:30:27.357077 25055 webserver.cc:460] Webserver started at http://127.15.222.126:46673/ using document root <none> and password file <none>
I20260809 03:30:27.357585 25055 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:27.357643 25055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:27.357867 25055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:27.359840 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "a9259360797e4f909ff385fe0628daf4"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "55588e372b7f603632a21353824d846f"
server_key_iv: "24e14e6394834eff461e088d8776f201"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.360376 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "a9259360797e4f909ff385fe0628daf4"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "55588e372b7f603632a21353824d846f"
server_key_iv: "24e14e6394834eff461e088d8776f201"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.364053 25055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:27.366264 25073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:27.367348 25055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:27.367465 25055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "a9259360797e4f909ff385fe0628daf4"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "55588e372b7f603632a21353824d846f"
server_key_iv: "24e14e6394834eff461e088d8776f201"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.367556 25055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:27.383018 25055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:27.383628 25055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:27.383803 25055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:27.390491 25055 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41045
I20260809 03:30:27.390545 25167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41045 every 8 connection(s)
I20260809 03:30:27.391430 25055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:27.394102 25168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:27.399258 25168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4: Bootstrap starting.
I20260809 03:30:27.400823 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 25055
I20260809 03:30:27.400956 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:27.401240 16249 external_mini_cluster.cc:1428] Setting key 7f72a41d01554a1c18883979a867ae45
I20260809 03:30:27.401862 25168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:27.402712 25168 log.cc:824] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:27.404480 25168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4: No bootstrap required, opened a new log
I20260809 03:30:27.407840 25168 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } }
I20260809 03:30:27.408052 25168 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:27.408109 25168 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9259360797e4f909ff385fe0628daf4, State: Initialized, Role: FOLLOWER
I20260809 03:30:27.408527 25168 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } }
I20260809 03:30:27.408661 25168 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:27.408756 25168 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:27.408842 25168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:27.409665 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.402286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.15.222.126:41045)
0809 03:30:27.402900 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.402916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.402949 (+ 33us) server_negotiation.cc:407] Connection header received
0809 03:30:27.403686 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.403705 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.404020 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.404316 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:27.404724 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.405641 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.406543 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.406991 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.407133 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.407909 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:27.407914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:27.407926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:27.408319 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:27.408375 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:27.408405 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:27.408488 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:27.408621 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 03:30:27.408996 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":129,"threads_started":1}
I20260809 03:30:27.409653 25168 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } }
I20260809 03:30:27.410018 25168 leader_election.cc:302] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9259360797e4f909ff385fe0628daf4; no voters:
I20260809 03:30:27.410320 25168 leader_election.cc:288] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:27.410401 25173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:27.410576 25173 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [term 1 LEADER]: Becoming Leader. State: Replica: a9259360797e4f909ff385fe0628daf4, State: Running, Role: LEADER
I20260809 03:30:27.410856 25173 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } }
I20260809 03:30:27.411535 25168 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:27.411975 25173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9259360797e4f909ff385fe0628daf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } } }
I20260809 03:30:27.412132 25173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:27.412249 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:27.412362 25173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9259360797e4f909ff385fe0628daf4. Latest consensus state: current_term: 1 leader_uuid: "a9259360797e4f909ff385fe0628daf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9259360797e4f909ff385fe0628daf4" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41045 } } }
I20260809 03:30:27.412438 25173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:27.412678 25184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:27.415717 25184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:27.420562 25184 catalog_manager.cc:1347] Generated new cluster ID: a003160b46724d02987cda00ebc84260
I20260809 03:30:27.420625 25184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:27.434087 25184 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:27.434873 25184 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:27.442507 25184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9259360797e4f909ff385fe0628daf4: Generated new TSK 0
I20260809 03:30:27.443226 25184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:27.458027 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.453583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.15.222.126:41045)
0809 03:30:27.453695 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.453699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.453854 (+ 155us) server_negotiation.cc:407] Connection header received
0809 03:30:27.454012 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.454014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.454059 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.454140 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:30:27.454349 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.454859 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.456035 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.456237 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.456300 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.456574 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:30:27.457646 (+ 1072us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 03:30:27.457895 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 03:30:27.458086 25171 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 03:30:27.458110 25215 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:27.453459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41045 (local address 127.0.0.1:36228)
0809 03:30:27.453770 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.453785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:27.453907 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:27.454152 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:27.454155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:27.454164 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:30:27.454267 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.454273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.454954 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.454958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.455935 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.455946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.456331 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.456335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.456418 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.456445 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:30:27.457809 (+ 1364us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 03:30:27.457952 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.222.126:41045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260809 03:30:27.458598 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 25055
E20260809 03:30:27.471050 25216 webserver.cc:533] Webserver: sslize failed:
E20260809 03:30:27.471112 25216 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T03:30:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T03:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:27Z Disabled control of system clock
I20260809 03:30:27.509608 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:45107
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:45107
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/testchainca.cert with env {}
W20260809 03:30:27.618297 25223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:27.618588 25223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:27.618649 25223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:27.622604 25223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:27.622658 25223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:27.622680 25223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:27.622701 25223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:27.622730 25223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 03:30:27.622753 25223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 03:30:27.622781 25223 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 03:30:27.627977 25223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:45107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:27.628964 25223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:27.629978 25223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:27.637800 25229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:27.637781 25235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:27.637993 25223 server_base.cc:1034] running on GCE node
W20260809 03:30:27.637814 25230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:27.638614 25223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:27.639613 25223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:27.640761 25223 hybrid_clock.cc:648] HybridClock initialized: now 1786246227640738 us; error 37 us; skew 500 ppm
I20260809 03:30:27.642442 25223 webserver.cc:460] Webserver started at http://127.15.222.126:46523/ using document root <none> and password file <none>
I20260809 03:30:27.642966 25223 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:27.643033 25223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:27.643209 25223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:27.645152 25223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "4af48fd74dd502c30a676a5a07409c3f"
server_key_iv: "bd575b1ee2b5c533551445a8d4f08ff9"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.645792 25223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "4af48fd74dd502c30a676a5a07409c3f"
server_key_iv: "bd575b1ee2b5c533551445a8d4f08ff9"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.649266 25223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:27.651508 25242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:27.652601 25223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:27.652712 25223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83"
format_stamp: "Formatted at 2026-08-09 03:30:27 on dist-test-slave-cvwc"
server_key: "4af48fd74dd502c30a676a5a07409c3f"
server_key_iv: "bd575b1ee2b5c533551445a8d4f08ff9"
server_key_version: "encryptionkey@0"
I20260809 03:30:27.652801 25223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:27.674819 25223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:27.675444 25223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:27.675607 25223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:27.682175 25223 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:45107
I20260809 03:30:27.682199 25333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:45107 every 8 connection(s)
I20260809 03:30:27.683099 25223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:27.684567 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 25223
I20260809 03:30:27.684655 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:27.684930 16249 external_mini_cluster.cc:1428] Setting key 60dea5fd67ff28e9204d40702d6ab615
I20260809 03:30:27.686460 25334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:27.693352 25334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83: Bootstrap starting.
I20260809 03:30:27.694980 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.686238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.15.222.126:45107)
0809 03:30:27.686958 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.686985 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.687036 (+ 51us) server_negotiation.cc:407] Connection header received
0809 03:30:27.687871 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.687894 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.688379 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.688729 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:27.689148 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.690593 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.691430 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.692100 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.692265 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.692970 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:27.692976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:27.692991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:27.693473 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:27.693547 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:27.693574 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:27.693668 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:27.693803 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:30:27.694061 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260809 03:30:27.695992 25334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:27.696820 25334 log.cc:824] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:27.697825 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:30:27.698918 25334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83: No bootstrap required, opened a new log
I20260809 03:30:27.702397 25334 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } }
I20260809 03:30:27.702617 25334 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:27.702673 25334 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a2c1eccce94b6c9268ca2d3cb72d83, State: Initialized, Role: FOLLOWER
I20260809 03:30:27.703059 25334 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } }
I20260809 03:30:27.703186 25334 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:27.703248 25334 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:27.703346 25334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:27.704191 25334 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } }
I20260809 03:30:27.704502 25334 leader_election.cc:302] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a2c1eccce94b6c9268ca2d3cb72d83; no voters:
I20260809 03:30:27.704782 25334 leader_election.cc:288] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:27.704910 25340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:27.705099 25340 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [term 1 LEADER]: Becoming Leader. State: Replica: b1a2c1eccce94b6c9268ca2d3cb72d83, State: Running, Role: LEADER
I20260809 03:30:27.705438 25340 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } }
I20260809 03:30:27.705924 25334 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:27.707054 25342 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } } }
I20260809 03:30:27.707144 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a2c1eccce94b6c9268ca2d3cb72d83. Latest consensus state: current_term: 1 leader_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a2c1eccce94b6c9268ca2d3cb72d83" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 45107 } } }
I20260809 03:30:27.707196 25342 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:27.707222 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:27.707787 25355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:27.710805 25355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:27.715489 25355 catalog_manager.cc:1347] Generated new cluster ID: f60cf6c3419e4547af48c3f5deb6a29d
I20260809 03:30:27.715548 25355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:27.721772 25355 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:27.722898 25355 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:27.727880 25355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1a2c1eccce94b6c9268ca2d3cb72d83: Generated new TSK 0
I20260809 03:30:27.728549 25355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:27.794819 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:27.738301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.15.222.126:45107)
0809 03:30:27.738424 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:27.738428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:27.738607 (+ 179us) server_negotiation.cc:407] Connection header received
0809 03:30:27.738748 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:27.738750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:27.738793 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:27.738872 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0809 03:30:27.739100 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.739573 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.740769 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.740969 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.741030 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.741315 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 03:30:27.794458 (+ 53143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 03:30:27.794706 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 03:30:27.794902 25338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 03:30:27.795403 25380 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:27.738240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:45107 (local address 127.0.0.1:53866)
0809 03:30:27.738518 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:27.738532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:27.738644 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:27.738886 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:27.738889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:27.738897 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 03:30:27.739012 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.739017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.739681 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.739684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.740665 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:27.740675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:27.741064 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:27.741068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:27.741160 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:27.741187 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 03:30:27.794708 (+ 53521us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 03:30:27.794932 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.222.126:45107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260809 03:30:27.795974 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 25223
2026-08-09T03:30:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (322 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:28 dist-test-slave-cvwc krb5kdc[25397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:28 dist-test-slave-cvwc krb5kdc[25397](info): set up 2 sockets
Aug 09 03:30:28 dist-test-slave-cvwc krb5kdc[25397](info): commencing operation
krb5kdc: starting...
W20260809 03:30:29.191658 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:29.305243 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46561
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:29.413137 25416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:29.413390 25416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:29.413444 25416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:29.417366 25416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:29.417418 25416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:29.417444 25416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:29.417464 25416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:29.417492 25416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:29.422593 25416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:29.423532 25416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:29.424504 25416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:29.432571 25429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:29.432621 25426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:29.432701 25416 server_base.cc:1034] running on GCE node
W20260809 03:30:29.432646 25431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:29.433354 25416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:29.434383 25416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:29.435555 25416 hybrid_clock.cc:648] HybridClock initialized: now 1786246229435528 us; error 35 us; skew 500 ppm
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:29.438180 25416 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:29.439002 25416 webserver.cc:460] Webserver started at http://127.15.222.126:46681/ using document root <none> and password file <none>
I20260809 03:30:29.439495 25416 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:29.439553 25416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:29.439724 25416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:29.441670 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "96b8209fd2c04194bc243bae10d408d6"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "857c55da5eaeb42fdda6eadcf39dad14"
server_key_iv: "049ba4142f4910bd0b3caddaa1849349"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.442217 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "96b8209fd2c04194bc243bae10d408d6"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "857c55da5eaeb42fdda6eadcf39dad14"
server_key_iv: "049ba4142f4910bd0b3caddaa1849349"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.445621 25416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260809 03:30:29.447849 25441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.448984 25416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:29.449095 25416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "96b8209fd2c04194bc243bae10d408d6"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "857c55da5eaeb42fdda6eadcf39dad14"
server_key_iv: "049ba4142f4910bd0b3caddaa1849349"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.449182 25416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:29.470918 25416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:29.473508 25416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:29.473675 25416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:29.480396 25416 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46561
I20260809 03:30:29.480425 25526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46561 every 8 connection(s)
I20260809 03:30:29.481297 25416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:29.484022 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:29.489329 25527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: Bootstrap starting.
I20260809 03:30:29.490588 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 25416
I20260809 03:30:29.490693 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:29.490952 16249 external_mini_cluster.cc:1428] Setting key af567ff074849e05f78cc0f6d9b7873e
I20260809 03:30:29.491928 25527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:29.492640 25527 log.cc:824] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:29.494709 25527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: No bootstrap required, opened a new log
I20260809 03:30:29.498718 25527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } }
I20260809 03:30:29.499001 25527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:29.499060 25527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8209fd2c04194bc243bae10d408d6, State: Initialized, Role: FOLLOWER
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:29.499485 25527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } }
I20260809 03:30:29.499625 25527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:29.499686 25527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:29.499778 25527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:29.500669 25527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } }
I20260809 03:30:29.501024 25527 leader_election.cc:302] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b8209fd2c04194bc243bae10d408d6; no voters:
I20260809 03:30:29.501330 25527 leader_election.cc:288] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:29.501425 25535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:29.501618 25535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [term 1 LEADER]: Becoming Leader. State: Replica: 96b8209fd2c04194bc243bae10d408d6, State: Running, Role: LEADER
I20260809 03:30:29.502003 25535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } }
I20260809 03:30:29.502523 25527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:29.503567 25537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b8209fd2c04194bc243bae10d408d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } } }
I20260809 03:30:29.503690 25537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:29.503983 25539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b8209fd2c04194bc243bae10d408d6. Latest consensus state: current_term: 1 leader_uuid: "96b8209fd2c04194bc243bae10d408d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b8209fd2c04194bc243bae10d408d6" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46561 } } }
I20260809 03:30:29.504115 25539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:29.505802 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.491989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.15.222.126:46561)
0809 03:30:29.492412 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:29.492429 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:29.492463 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:29.493137 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:29.493161 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:29.493536 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:29.493867 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:29.494672 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.495618 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.496749 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.497188 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.497294 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:29.500035 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:29.500040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:29.500052 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:29.500077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:29.501817 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.502311 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.502317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.502322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.502477 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.502781 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.502784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.502786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.503282 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:29.504254 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:29.504418 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 03:30:29.504885 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
W20260809 03:30:29.507467 25559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:29.507542 25559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:29.507611 25549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:29.508421 25549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:29.513423 25549 catalog_manager.cc:1347] Generated new cluster ID: e3a4161badd040f7904cfde7e2552f76
I20260809 03:30:29.513484 25549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:29.521245 25549 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:29.522115 25549 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:29.530617 25549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b8209fd2c04194bc243bae10d408d6: Generated new TSK 0
I20260809 03:30:29.531317 25549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:29.564232 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:29.676033 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:29.676307 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:29.676370 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:29.680465 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:29.680524 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:29.680600 25568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:29.685830 25568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:29.686887 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:29.687877 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:29.697069 25576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:29.697094 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:29.697077 25575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:29.697289 25568 server_base.cc:1034] running on GCE node
I20260809 03:30:29.697817 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:29.698426 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:29.699605 25568 hybrid_clock.cc:648] HybridClock initialized: now 1786246229699590 us; error 31 us; skew 500 ppm
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:29.709352 25568 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:29.710304 25568 webserver.cc:460] Webserver started at http://127.15.222.65:46231/ using document root <none> and password file <none>
I20260809 03:30:29.710884 25568 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:29.710955 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:29.711161 25568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:29.713186 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "74f86e47554c409d8c17f93b7d42f441"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "6e515e13ab5c3bdefcac914aed18af3d"
server_key_iv: "14482238949693addf25b78678fde1ca"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.713773 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "74f86e47554c409d8c17f93b7d42f441"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "6e515e13ab5c3bdefcac914aed18af3d"
server_key_iv: "14482238949693addf25b78678fde1ca"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.717272 25568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:29.719558 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.720682 25568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:29.720794 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "74f86e47554c409d8c17f93b7d42f441"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "6e515e13ab5c3bdefcac914aed18af3d"
server_key_iv: "14482238949693addf25b78678fde1ca"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.720880 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:29.733975 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:29.736474 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:29.736637 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:29.737188 25568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:29.738000 25568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:29.738047 25568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.738099 25568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:29.738124 25568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.746500 25568 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:35595
I20260809 03:30:29.746521 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:35595 every 8 connection(s)
I20260809 03:30:29.747306 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:29.751262 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 25568
I20260809 03:30:29.751390 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:29.751688 16249 external_mini_cluster.cc:1428] Setting key 447b7439817611f4d686bb60c7328517
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:29.760206 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.748979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:45981 (local address 127.15.222.126:46561)
0809 03:30:29.749098 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:29.749101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:29.749886 (+ 785us) server_negotiation.cc:407] Connection header received
0809 03:30:29.751267 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:29.751270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:29.751330 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:29.751427 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:29.752648 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.753289 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.754416 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.754652 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.754739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:29.756974 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:29.756978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:29.756980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:29.757004 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:29.758333 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.758806 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.758809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.758810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.758885 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.759187 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.759190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.759192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.759363 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:29.759456 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:29.759913 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 03:30:29.760043 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:29.760964 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.749322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46561 (local address 127.15.222.65:45981)
0809 03:30:29.749756 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:29.749796 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:29.751037 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:29.751594 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:29.751601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:29.751966 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:29.752510 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:29.752524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.753410 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.753413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:29.754294 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:29.754303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.754784 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.754787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:29.754947 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:29.755469 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:29.755485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:29.756817 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:29.758442 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:29.758448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:29.758462 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:29.758715 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:29.758967 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:29.758970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:29.758972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:29.759102 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:29.759466 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:29.759471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:29.759698 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 03:30:29.759855 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 03:30:29.760165 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260809 03:30:29.762099 25795 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46561
I20260809 03:30:29.762300 25795 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:29.762768 25795 heartbeater.cc:511] Master 127.15.222.126:46561 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:29.764262 25472 ts_manager.cc:194] Registered new tserver with Master: 74f86e47554c409d8c17f93b7d42f441 (127.15.222.65:35595)
I20260809 03:30:29.765748 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:45981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:29.776401 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:29.884780 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:29.885008 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:29.885046 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:29.888962 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:29.889014 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:29.889067 25804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:29.894073 25804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:29.894981 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:29.895882 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:29.905005 25809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:29.905103 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:29.905005 25810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:29.905179 25804 server_base.cc:1034] running on GCE node
I20260809 03:30:29.905627 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:29.906232 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:29.907384 25804 hybrid_clock.cc:648] HybridClock initialized: now 1786246229907371 us; error 24 us; skew 500 ppm
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:29.909951 25804 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:29.910764 25804 webserver.cc:460] Webserver started at http://127.15.222.66:36329/ using document root <none> and password file <none>
I20260809 03:30:29.911290 25804 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:29.911352 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:29.911573 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:29.913545 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "cc699146dec34962b527a81127c956da"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "cbf6aad0ad8346f4713cce670c2a5735"
server_key_iv: "7229e2d55ebdf4ad71777303885213e9"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.914203 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "cc699146dec34962b527a81127c956da"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "cbf6aad0ad8346f4713cce670c2a5735"
server_key_iv: "7229e2d55ebdf4ad71777303885213e9"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.917671 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:29.919979 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.921113 25804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:29.921224 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "cc699146dec34962b527a81127c956da"
format_stamp: "Formatted at 2026-08-09 03:30:29 on dist-test-slave-cvwc"
server_key: "cbf6aad0ad8346f4713cce670c2a5735"
server_key_iv: "7229e2d55ebdf4ad71777303885213e9"
server_key_version: "encryptionkey@0"
I20260809 03:30:29.921311 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:29.941454 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:29.944430 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:29.944595 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:29.945138 25804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:29.945955 25804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:29.946004 25804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.946058 25804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:29.946084 25804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:29.954406 25804 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:37287
I20260809 03:30:29.954453 26037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:37287 every 8 connection(s)
I20260809 03:30:29.955232 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:29.962384 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 25804
I20260809 03:30:29.962509 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:29.962787 16249 external_mini_cluster.cc:1428] Setting key e1dc80fa87a96cde5b16e44d26007d1f
Aug 09 03:30:29 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:29.968029 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.956846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:57455 (local address 127.15.222.126:46561)
0809 03:30:29.956976 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:29.956980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:29.957806 (+ 826us) server_negotiation.cc:407] Connection header received
0809 03:30:29.959084 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:29.959087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:29.959132 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:29.959208 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:29.960231 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.960739 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.961851 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.962070 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.962159 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:29.964587 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:29.964591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:29.964595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:29.964620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:29.966094 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.966650 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.966652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.966654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.966725 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:29.967060 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:29.967062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:29.967063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:29.967220 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:29.967286 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:29.967824 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 03:30:29.967913 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:29.969046 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:29.957224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46561 (local address 127.15.222.66:57455)
0809 03:30:29.957651 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:29.957689 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:29.958904 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:29.959346 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:29.959352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:29.959680 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:29.960121 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:29.960134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.960822 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.960824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:29.961734 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:29.961746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:29.962215 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:29.962220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:29.962423 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:29.962948 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:29.962964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:29.964429 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:29.966203 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:29.966209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:29.966224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:29.966563 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:29.966798 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:29.966800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:29.966803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:29.966980 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:29.967300 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:29.967305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:29.967599 (+ 294us) client_negotiation.cc:769] Sending connection context
0809 03:30:29.967801 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 03:30:29.968116 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260809 03:30:29.970152 26038 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46561
I20260809 03:30:29.970402 26038 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:29.970973 26038 heartbeater.cc:511] Master 127.15.222.126:46561 requested a full tablet report, sending...
I20260809 03:30:29.972026 25472 ts_manager.cc:194] Registered new tserver with Master: cc699146dec34962b527a81127c956da (127.15.222.66:37287)
I20260809 03:30:29.972558 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:57455
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:29.989828 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:30.100513 26045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:30.100744 26045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:30.100781 26045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:30.104712 26045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:30.104763 26045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:30.104817 26045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:30.109858 26045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:30.110756 26045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:30.111713 26045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:30.120911 26045 server_base.cc:1034] running on GCE node
W20260809 03:30:30.120819 26065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:30.120824 26059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:30.120819 26060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:30.121469 26045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:30.122080 26045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:30.123231 26045 hybrid_clock.cc:648] HybridClock initialized: now 1786246230123216 us; error 27 us; skew 500 ppm
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.126011 26045 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:30.126786 26045 webserver.cc:460] Webserver started at http://127.15.222.67:46283/ using document root <none> and password file <none>
I20260809 03:30:30.127300 26045 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:30.127364 26045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:30.127548 26045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:30.129534 26045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "538a2c6b582b4cc8a7017b0d63716b67"
format_stamp: "Formatted at 2026-08-09 03:30:30 on dist-test-slave-cvwc"
server_key: "6c09e214d33551b0e8473d813c98e2d5"
server_key_iv: "9a95470d2c20e012f756ea65aa1bf730"
server_key_version: "encryptionkey@0"
I20260809 03:30:30.130106 26045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "538a2c6b582b4cc8a7017b0d63716b67"
format_stamp: "Formatted at 2026-08-09 03:30:30 on dist-test-slave-cvwc"
server_key: "6c09e214d33551b0e8473d813c98e2d5"
server_key_iv: "9a95470d2c20e012f756ea65aa1bf730"
server_key_version: "encryptionkey@0"
I20260809 03:30:30.133471 26045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:30.135739 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:30.136839 26045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:30.136950 26045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "538a2c6b582b4cc8a7017b0d63716b67"
format_stamp: "Formatted at 2026-08-09 03:30:30 on dist-test-slave-cvwc"
server_key: "6c09e214d33551b0e8473d813c98e2d5"
server_key_iv: "9a95470d2c20e012f756ea65aa1bf730"
server_key_version: "encryptionkey@0"
I20260809 03:30:30.137035 26045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:30.148730 26045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:30.151134 26045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:30.151284 26045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:30.151783 26045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:30.152592 26045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:30.152638 26045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:30.152686 26045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:30.152716 26045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:30.160702 26045 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:39423
I20260809 03:30:30.160732 26282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:39423 every 8 connection(s)
I20260809 03:30:30.161476 26045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:30.165839 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 26045
I20260809 03:30:30.165953 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:30.166235 16249 external_mini_cluster.cc:1428] Setting key 4623c83ef91f7b9ac26d17ab16b2c8ff
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:30.173982 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.163083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:49467 (local address 127.15.222.126:46561)
0809 03:30:30.163217 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.163222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.164008 (+ 786us) server_negotiation.cc:407] Connection header received
0809 03:30:30.165246 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.165248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.165296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.165365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.166758 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.167452 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.168675 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.168884 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.168964 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:30.171175 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:30.171177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:30.171179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:30.171204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:30.172365 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.172782 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.172784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.172785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.172852 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.173130 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.173132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.173133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.173289 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:30.173356 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:30.173776 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 03:30:30.173866 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:30:30.174721 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.163441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46561 (local address 127.15.222.67:49467)
0809 03:30:30.163870 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:30.163911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:30.165067 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:30.165519 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:30.165527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:30.166007 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:30.166614 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:30.166631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.167569 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.167572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:30.168568 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:30.168578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.168994 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.168996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:30.169127 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:30.169632 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:30.169648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:30.171021 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:30.172468 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:30.172472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:30.172483 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:30.172711 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:30.172934 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:30.172936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:30.172937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:30.173067 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:30.173355 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:30.173360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:30.173575 (+ 215us) client_negotiation.cc:769] Sending connection context
0809 03:30:30.173740 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 03:30:30.173987 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
I20260809 03:30:30.175664 26284 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46561
I20260809 03:30:30.175870 26284 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:30.176297 26284 heartbeater.cc:511] Master 127.15.222.126:46561 requested a full tablet report, sending...
I20260809 03:30:30.177217 25472 ts_manager.cc:194] Registered new tserver with Master: 538a2c6b582b4cc8a7017b0d63716b67 (127.15.222.67:39423)
I20260809 03:30:30.177776 25472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:49467
I20260809 03:30:30.179656 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/test-admin.keytab.
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.194856 16249 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 03:30:30.197602 26291 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:30.196229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.15.222.65:35595)
0809 03:30:30.196602 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.196611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.196635 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:30:30.196684 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.196690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.196875 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.196997 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.197415 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41176: BlockingRecv error: recv got EOF from 127.0.0.1:41176 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":177,"threads_started":1}
W20260809 03:30:30.197652 16249 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc' of type FILE
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.198632 16249 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.199486 16249 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:30.208010 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.200241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.15.222.65:35595)
0809 03:30:30.200365 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.200368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.200408 (+ 40us) server_negotiation.cc:407] Connection header received
0809 03:30:30.200525 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.200528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.200562 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.200625 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.201056 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.201845 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.202634 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.202995 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.203122 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:30.205157 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:30.205165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:30.205179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:30.205199 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:30.206590 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.206957 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.206962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.206963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.207036 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.207212 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.207214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.207215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.207420 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:30.207570 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:30.207703 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 03:30:30.207838 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 03:30:30.211382 26291 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:30.210576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.15.222.65:35595)
0809 03:30:30.210680 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.210684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.210769 (+ 85us) server_negotiation.cc:407] Connection header received
0809 03:30:30.210861 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.210864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.210901 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.210960 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.211309 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41194: BlockingRecv error: recv got EOF from 127.0.0.1:41194 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260809 03:30:30.211664 16249 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc)
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.212512 16249 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.213266 16249 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:30.221544 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.213966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.15.222.65:35595)
0809 03:30:30.214065 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.214069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.214164 (+ 95us) server_negotiation.cc:407] Connection header received
0809 03:30:30.214237 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.214239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.214271 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.214328 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.214720 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.215198 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.215998 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.216317 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.216425 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:30.218931 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:30.218935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:30.218938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:30.218961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:30.220166 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.220530 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.220533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.220535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.220603 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.220836 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.220839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.220840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.220989 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:30.221088 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:30.221320 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 03:30:30.221429 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 03:30:30.223872 26291 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:30.222947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.15.222.65:35595)
0809 03:30:30.223055 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.223059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.223120 (+ 61us) server_negotiation.cc:407] Connection header received
0809 03:30:30.223222 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.223225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.223259 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.223322 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.223792 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41226: BlockingRecv error: recv got EOF from 127.0.0.1:41226 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 03:30:30.224000 16249 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786246200571663-16249-0/krb5kdc/krb5cc' of type FILE
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.224846 16249 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:30.225651 16249 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[25397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:30:30.232676 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:30.226367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.15.222.65:35595)
0809 03:30:30.226472 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:30.226476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:30.226544 (+ 68us) server_negotiation.cc:407] Connection header received
0809 03:30:30.226647 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:30.226650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:30.226684 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:30.226742 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:30.227193 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.227645 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.228363 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:30.228568 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:30.228647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:30.230494 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:30.230496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:30.230498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:30.230518 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:30.231656 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.231952 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.231954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.231955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.232014 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:30.232173 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:30.232174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:30.232175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:30.232293 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:30.232369 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:30.232514 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 03:30:30.232587 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:30:30.233444 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 25568
I20260809 03:30:30.246579 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 25804
I20260809 03:30:30.253621 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 26045
I20260809 03:30:30.259781 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 25416
2026-08-09T03:30:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2214 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[26311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[26311](info): set up 2 sockets
Aug 09 03:30:30 dist-test-slave-cvwc krb5kdc[26311](info): commencing operation
krb5kdc: starting...
W20260809 03:30:31.397050 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:31 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab.
Entry for principal oryx/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab.
I20260809 03:30:31.509644 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41409
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab
--principal=oryx/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:31.618947 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:31.619192 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:31.619230 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:31.623101 26333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:31.623149 26333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:31.623167 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:31.623181 26333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:31.623195 26333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:31.628227 26333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab
--principal=oryx/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:31.629151 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:31.630095 26333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:31.638162 26347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:31.638268 26345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:31.638183 26349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:31.638284 26333 server_base.cc:1034] running on GCE node
I20260809 03:30:31.639030 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:31.640036 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:31.641194 26333 hybrid_clock.cc:648] HybridClock initialized: now 1786246231641176 us; error 31 us; skew 500 ppm
Aug 09 03:30:31 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:31.643751 26333 init.cc:377] Logged in from keytab as oryx/127.15.222.126@KRBTEST.COM (short username oryx)
I20260809 03:30:31.644587 26333 webserver.cc:460] Webserver started at http://127.15.222.126:46865/ using document root <none> and password file <none>
I20260809 03:30:31.645102 26333 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:31.645169 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:31.645364 26333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:31.647370 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "dcf35202e96c43009c3150248750bfad"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "59668115494f58465bba5c74adaf8d39"
server_key_iv: "a46ef0262b36089961cc71c72219189d"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.647907 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "dcf35202e96c43009c3150248750bfad"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "59668115494f58465bba5c74adaf8d39"
server_key_iv: "a46ef0262b36089961cc71c72219189d"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.651469 26333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:31.653815 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:31.654953 26333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:31.655072 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "dcf35202e96c43009c3150248750bfad"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "59668115494f58465bba5c74adaf8d39"
server_key_iv: "a46ef0262b36089961cc71c72219189d"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.655160 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:31.665553 26333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:31.668316 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:31.668501 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:31.675595 26333 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41409
I20260809 03:30:31.675843 26445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41409 every 8 connection(s)
I20260809 03:30:31.676599 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:31.679432 26446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:31.684844 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Bootstrap starting.
I20260809 03:30:31.685180 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 26333
I20260809 03:30:31.685309 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:31.685613 16249 external_mini_cluster.cc:1428] Setting key 734cab3f6365726c7190765e8785a713
I20260809 03:30:31.687529 26446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:31.688236 26446 log.cc:824] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:31.690323 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: No bootstrap required, opened a new log
Aug 09 03:30:31 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:31.694574 26446 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } }
I20260809 03:30:31.694797 26446 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:31.694865 26446 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf35202e96c43009c3150248750bfad, State: Initialized, Role: FOLLOWER
I20260809 03:30:31.695307 26446 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } }
I20260809 03:30:31.695442 26446 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:31.695506 26446 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:31.695600 26446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:31.696390 26446 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } }
I20260809 03:30:31.696719 26446 leader_election.cc:302] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcf35202e96c43009c3150248750bfad; no voters:
I20260809 03:30:31.697022 26446 leader_election.cc:288] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:31.697103 26458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:31.697340 26458 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 LEADER]: Becoming Leader. State: Replica: dcf35202e96c43009c3150248750bfad, State: Running, Role: LEADER
I20260809 03:30:31.697685 26458 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } }
I20260809 03:30:31.698340 26446 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:31.699412 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf35202e96c43009c3150248750bfad. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } }
I20260809 03:30:31.699534 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:31.699710 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.686928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.15.222.126:41409)
0809 03:30:31.687316 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:31.687332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:31.687368 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:30:31.688015 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:31.688034 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:31.688371 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:31.688699 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:31.689460 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.690481 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.691589 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.692075 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.692190 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:31.694880 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:31.694887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:31.694904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:31.694937 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:31.696566 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:31.696978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:31.696985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:31.696990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:31.697121 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:31.697411 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:31.697414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:31.697416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:31.697943 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:31.698138 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:31.698399 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 03:30:31.698722 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260809 03:30:31.699802 26459 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } }
I20260809 03:30:31.699872 26459 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:31.699927 26469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:31.704212 26469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:31.709596 26469 catalog_manager.cc:1347] Generated new cluster ID: f4f4b7b1086a4ad0a66cc5856b5a7c7a
I20260809 03:30:31.709661 26469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:31.719228 26469 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:31.720258 26469 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:31.725162 26469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Generated new TSK 0
I20260809 03:30:31.725780 26469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
I20260809 03:30:31.756196 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:31.864637 26485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:31.864868 26485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:31.864908 26485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:31.868830 26485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:31.868884 26485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:31.868950 26485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:31.874043 26485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:31.874980 26485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:31.875929 26485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:31.884948 26492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:31.884948 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:31.884958 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:31.885193 26485 server_base.cc:1034] running on GCE node
I20260809 03:30:31.885696 26485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:31.886307 26485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:31.887462 26485 hybrid_clock.cc:648] HybridClock initialized: now 1786246231887441 us; error 32 us; skew 500 ppm
Aug 09 03:30:31 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:31.890045 26485 init.cc:377] Logged in from keytab as oryx/127.15.222.65@KRBTEST.COM (short username oryx)
I20260809 03:30:31.890828 26485 webserver.cc:460] Webserver started at http://127.15.222.65:41829/ using document root <none> and password file <none>
I20260809 03:30:31.891328 26485 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:31.891400 26485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:31.891604 26485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:31.893555 26485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "001906698b2441979f3f786812886e37"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "7556661841d757723c5b64b631ee6b0d"
server_key_iv: "2f2b179c30b36897a4c83d0fb92c694e"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.894120 26485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "001906698b2441979f3f786812886e37"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "7556661841d757723c5b64b631ee6b0d"
server_key_iv: "2f2b179c30b36897a4c83d0fb92c694e"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.897538 26485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:31.899863 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:31.900982 26485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:31.901095 26485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "001906698b2441979f3f786812886e37"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "7556661841d757723c5b64b631ee6b0d"
server_key_iv: "2f2b179c30b36897a4c83d0fb92c694e"
server_key_version: "encryptionkey@0"
I20260809 03:30:31.901181 26485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:31.923748 26485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:31.926715 26485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:31.926889 26485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:31.927436 26485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:31.928215 26485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:31.928261 26485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:31.928313 26485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:31.928341 26485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:31.936503 26485 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44941
I20260809 03:30:31.936553 26708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44941 every 8 connection(s)
I20260809 03:30:31.937325 26485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:31.942233 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 26485
I20260809 03:30:31.942318 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:31.942621 16249 external_mini_cluster.cc:1428] Setting key 5f7c4c326bfd7d5816714e9c1bc44127
Aug 09 03:30:31 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:31.950389 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.938986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:36717 (local address 127.15.222.126:41409)
0809 03:30:31.939122 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:31.939126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:31.939934 (+ 808us) server_negotiation.cc:407] Connection header received
0809 03:30:31.941200 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:31.941203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:31.941248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:31.941318 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:31.942491 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.943151 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.944307 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.944546 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.944631 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:31.947128 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:31.947132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:31.947134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:31.947162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:31.948524 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:31.948975 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:31.948979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:31.948981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:31.949060 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:31.949356 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:31.949359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:31.949361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:31.949536 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:31.949623 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:31.950105 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 03:30:31.950225 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:31.951099 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:31.939354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:36717)
0809 03:30:31.939792 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:31.939832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:31.941014 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:31.941451 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:31.941457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:31.941828 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:31.942287 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:31.942305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.943269 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.943272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:31.944196 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:31.944206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:31.944678 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:31.944682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:31.944836 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:31.945346 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:31.945362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:31.946957 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:31.948632 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:31.948636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:31.948647 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:31.948875 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:31.949141 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:31.949144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:31.949145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:31.949269 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:31.949634 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:31.949640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:31.949889 (+ 249us) client_negotiation.cc:769] Sending connection context
0809 03:30:31.950048 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 03:30:31.950340 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260809 03:30:31.952219 26709 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:31.952462 26709 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:31.952984 26709 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:31.954365 26387 ts_manager.cc:194] Registered new tserver with Master: 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:31.955621 26387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:36717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
I20260809 03:30:31.968410 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:32.077247 26716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:32.077482 26716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:32.077522 26716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:32.081370 26716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:32.081421 26716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:32.081482 26716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:32.086599 26716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:32.087541 26716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:32.088490 26716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:32.097515 26727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.097522 26724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.097549 26723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.097834 26716 server_base.cc:1034] running on GCE node
I20260809 03:30:32.098306 26716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:32.098881 26716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:32.100035 26716 hybrid_clock.cc:648] HybridClock initialized: now 1786246232100017 us; error 38 us; skew 500 ppm
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:32.102659 26716 init.cc:377] Logged in from keytab as oryx/127.15.222.66@KRBTEST.COM (short username oryx)
I20260809 03:30:32.103466 26716 webserver.cc:460] Webserver started at http://127.15.222.66:39983/ using document root <none> and password file <none>
I20260809 03:30:32.104004 26716 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.104070 26716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:32.104282 26716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:32.106351 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "2d864ffd4065481ead0362cc6fc23f3d"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "d33f94e51e1aa677d283c4e3203c3542"
server_key_iv: "f021b8306d2b57d797d1a459fe8d4754"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.106879 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "2d864ffd4065481ead0362cc6fc23f3d"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "d33f94e51e1aa677d283c4e3203c3542"
server_key_iv: "f021b8306d2b57d797d1a459fe8d4754"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.110371 26716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:32.112565 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.113698 26716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:32.113853 26716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "2d864ffd4065481ead0362cc6fc23f3d"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "d33f94e51e1aa677d283c4e3203c3542"
server_key_iv: "f021b8306d2b57d797d1a459fe8d4754"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.113938 26716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.127600 26716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:32.130785 26716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:32.130973 26716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:32.131553 26716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:32.132411 26716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:32.132464 26716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.132520 26716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:32.132553 26716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.141470 26716 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42195
I20260809 03:30:32.141492 26916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42195 every 8 connection(s)
I20260809 03:30:32.142382 26716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:32.144503 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 26716
I20260809 03:30:32.144590 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:32.144848 16249 external_mini_cluster.cc:1428] Setting key f915becf34308c5df8a9eec90a161f68
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:32.155858 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.144095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46613 (local address 127.15.222.126:41409)
0809 03:30:32.144223 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.144226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.145037 (+ 811us) server_negotiation.cc:407] Connection header received
0809 03:30:32.146561 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.146564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.146614 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.146703 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.147767 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.148395 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.149861 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.150150 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.150259 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.152491 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.152494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.152496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.152518 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.154057 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.154509 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.154512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.154514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.154602 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.154875 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.154878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.154879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.155113 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.155208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.155638 (+ 430us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.155740 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:30:32.156651 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.144424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:46613)
0809 03:30:32.144881 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.144932 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.146354 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.146833 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.146840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.147163 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.147628 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.147641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.148521 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.148524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.149742 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.149754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.150283 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.150285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.150430 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.150954 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.150970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.152342 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.154159 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.154163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.154174 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.154412 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.154682 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.154684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.154686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.154800 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.155214 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.155219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.155438 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.155583 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.155812 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.67@KRBTEST.COM" created.
I20260809 03:30:32.157886 26917 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:32.158157 26917 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:32.158726 26917 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:32.159816 26387 ts_manager.cc:194] Registered new tserver with Master: 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:32.160378 26387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:46613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
I20260809 03:30:32.170639 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:32.283097 26924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:32.283324 26924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:32.283362 26924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:32.287182 26924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:32.287233 26924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:32.287285 26924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:32.292265 26924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:32.293148 26924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:32.294199 26924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:32.303680 26940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.303689 26938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.303689 26936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.303902 26924 server_base.cc:1034] running on GCE node
I20260809 03:30:32.304457 26924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:32.305033 26924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:32.306181 26924 hybrid_clock.cc:648] HybridClock initialized: now 1786246232306154 us; error 46 us; skew 500 ppm
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:32.311180 26924 init.cc:377] Logged in from keytab as oryx/127.15.222.67@KRBTEST.COM (short username oryx)
I20260809 03:30:32.311959 26924 webserver.cc:460] Webserver started at http://127.15.222.67:33339/ using document root <none> and password file <none>
I20260809 03:30:32.312458 26924 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.312512 26924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:32.312675 26924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:32.314590 26924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "557595f6ed2d40b9840fd838e7feadf9"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "388cc52336606ed952466f7e55dbae3b"
server_key_iv: "ec0e07733ba4d4d34224208324debb74"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.315088 26924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "557595f6ed2d40b9840fd838e7feadf9"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "388cc52336606ed952466f7e55dbae3b"
server_key_iv: "ec0e07733ba4d4d34224208324debb74"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.318670 26924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:32.320869 26948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.322068 26924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:32.322165 26924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "557595f6ed2d40b9840fd838e7feadf9"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "388cc52336606ed952466f7e55dbae3b"
server_key_iv: "ec0e07733ba4d4d34224208324debb74"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.322256 26924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.336706 26924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:32.339224 26924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:32.339389 26924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:32.339917 26924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:32.340703 26924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:32.340747 26924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.340788 26924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:32.340816 26924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.348802 26924 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:37765
I20260809 03:30:32.348827 27124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:37765 every 8 connection(s)
I20260809 03:30:32.349611 26924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:32.358716 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 26924
I20260809 03:30:32.358814 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:32.359131 16249 external_mini_cluster.cc:1428] Setting key 12a6ef091c4a44f3786c45547ff18411
I20260809 03:30:32.361908 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.351244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37431 (local address 127.15.222.126:41409)
0809 03:30:32.351360 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.351364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.352147 (+ 783us) server_negotiation.cc:407] Connection header received
0809 03:30:32.353382 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.353386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.353433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.353503 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.354579 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.355092 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.356105 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.356299 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.356363 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.358602 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.358604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.358606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.358629 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.360034 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.360542 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.360544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.360546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.360620 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.360956 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.360958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.360959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.361156 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.361243 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.361671 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.361792 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:30:32.362702 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.351559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:37431)
0809 03:30:32.351992 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.352032 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.353202 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.353649 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.353655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.354001 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.354460 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.354474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.355178 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.355180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.356017 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.356026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.356408 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.356411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.356544 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.357047 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.357063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.358433 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.360135 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.360140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.360150 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.360460 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.360694 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.360697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.360699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.360863 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.361259 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.361265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.361482 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.361630 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.361917 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":128,"threads_started":1}
I20260809 03:30:32.363696 27128 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:32.363904 27128 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:32.364331 27128 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:32.365226 26387 ts_manager.cc:194] Registered new tserver with Master: 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765)
I20260809 03:30:32.365797 26387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:37431
I20260809 03:30:32.372579 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.125.keytab.
Entry for principal oryx/127.15.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.125.keytab.
I20260809 03:30:32.397480 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:41829
--webserver_interface=127.15.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.125.keytab
--principal=oryx/127.15.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41409,127.15.222.125:41829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:32.507109 27137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:32.507437 27137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:32.507555 27137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:32.513093 27137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:32.513165 27137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:32.513190 27137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:32.513212 27137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:32.513242 27137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:32.520298 27137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41409,127.15.222.125:41829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.125.keytab
--principal=oryx/127.15.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:41829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
W20260809 03:30:32.520516 27137 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.222.126:41409,127.15.222.125:41829'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 03:30:32.521627 27137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:32.523010 27137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:32.533819 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.533828 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.533851 27146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.534389 27137 server_base.cc:1034] running on GCE node
I20260809 03:30:32.535025 27137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:32.536455 27137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:32.537654 27137 hybrid_clock.cc:648] HybridClock initialized: now 1786246232537612 us; error 56 us; skew 500 ppm
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:32.541002 27137 init.cc:377] Logged in from keytab as oryx/127.15.222.125@KRBTEST.COM (short username oryx)
I20260809 03:30:32.542125 27137 webserver.cc:460] Webserver started at http://127.15.222.125:40879/ using document root <none> and password file <none>
I20260809 03:30:32.542819 27137 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.542905 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:32.543143 27137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:32.545647 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/instance:
uuid: "4c379d4bee144e8babd3ea195e61c613"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e2bf3590128812a71c5b105ab7109b1e"
server_key_iv: "f429562859f22118e75ce297106a4e39"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.546344 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal/instance:
uuid: "4c379d4bee144e8babd3ea195e61c613"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e2bf3590128812a71c5b105ab7109b1e"
server_key_iv: "f429562859f22118e75ce297106a4e39"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.550920 27137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 03:30:32.553416 27164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.554932 27137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 03:30:32.555052 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "4c379d4bee144e8babd3ea195e61c613"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e2bf3590128812a71c5b105ab7109b1e"
server_key_iv: "f429562859f22118e75ce297106a4e39"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.555157 27137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.576921 27137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:32.579671 27137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:32.579847 27137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:32.586612 27137 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.125:41829
I20260809 03:30:32.586869 27242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.125:41829 every 8 connection(s)
I20260809 03:30:32.587579 27137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
I20260809 03:30:32.590276 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:32.593200 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 27137
I20260809 03:30:32.594142 27244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.125@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:32.605582 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.595847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.15.222.126:41409)
0809 03:30:32.595984 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.595988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.596278 (+ 290us) server_negotiation.cc:407] Connection header received
0809 03:30:32.596436 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.596440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.596488 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.596530 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.597360 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.598182 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.599266 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.599490 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.599578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.601930 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.601934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.601937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.601964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.603377 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.603960 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.603963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.603965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.604054 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.604372 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.604375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.604376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.604672 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.604754 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.605292 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.605418 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:32.606499 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.595749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:36690)
0809 03:30:32.596143 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.596188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.596315 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.596682 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.596686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.597009 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.597233 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.597243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.598297 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.598300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.599148 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.599158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.599622 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.599625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.599770 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.600309 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.600329 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.601791 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.603486 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.603493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.603502 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.603887 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.604136 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.604138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.604140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.604282 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.604767 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.604773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.605072 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.605234 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.605528 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":20,"spinlock_wait_cycles":4096,"thread_start_us":125,"threads_started":1}
I20260809 03:30:32.606499 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.594686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.15.222.125:41829)
0809 03:30:32.595082 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.595098 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.595132 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:32.595755 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.595777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.596137 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.596395 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.597163 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.598410 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.599281 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.599633 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.599775 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.602332 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.602337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.602346 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.602366 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.603823 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.604268 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.604273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.604275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.604348 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.604588 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.604590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.604591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.605072 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.605246 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.605458 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.605571 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":10880,"thread_start_us":119,"threads_started":1}
I20260809 03:30:32.607728 27244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.125@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:32.614763 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.608144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:58708)
0809 03:30:32.608288 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.608300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.608369 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.608563 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.608566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.608683 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.608769 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.608773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.609369 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.609371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.610524 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.610534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.610902 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.610904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.610968 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.611285 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.611294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.612331 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.613554 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.613557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.613559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.613742 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.613957 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.613960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.613962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.614032 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.614327 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.614330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.614407 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.614487 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.614605 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 03:30:32.614782 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.608200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.15.222.125:41829)
0809 03:30:32.608298 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.608301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.608440 (+ 139us) server_negotiation.cc:407] Connection header received
0809 03:30:32.608459 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.608460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.608484 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.608528 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.608848 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.609299 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.610611 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.610822 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.610892 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.612443 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.612446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.612449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.612471 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.613467 (+ 996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.613813 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.613816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.613818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.613884 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.614101 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.614104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.614106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.614248 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.614326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.614537 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.614657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":42}
I20260809 03:30:32.617375 27244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Bootstrap starting.
I20260809 03:30:32.619659 27244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:32.620337 27244 log.cc:824] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:32.622103 27244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: No bootstrap required, opened a new log
I20260809 03:30:32.624994 27244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } }
I20260809 03:30:32.625187 27244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:32.625243 27244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c379d4bee144e8babd3ea195e61c613, State: Initialized, Role: FOLLOWER
I20260809 03:30:32.625670 27244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } }
I20260809 03:30:32.626163 27251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } } }
I20260809 03:30:32.626268 27251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:32.626845 27244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 03:30:32.631122 27264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:32.631189 27264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:32.633783 27137 master_runner.cc:428] Detected that this master 4c379d4bee144e8babd3ea195e61c613 is joining an existing cluster
I20260809 03:30:32.633836 27137 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.222.125:41829
I20260809 03:30:32.643291 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.635696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.15.222.126:41409)
0809 03:30:32.635807 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.635811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.635962 (+ 151us) server_negotiation.cc:407] Connection header received
0809 03:30:32.636066 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.636069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.636118 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.636185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.636806 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.637465 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.638741 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.639062 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.639148 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.640288 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.640292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.640300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.640324 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.641668 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642139 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.642142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.642144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.642223 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642462 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.642465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.642467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.642637 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.642721 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.643055 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.643159 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:30:32.643303 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.635620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:36706)
0809 03:30:32.635881 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.635901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.635993 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.636305 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.636308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.636538 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.636674 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.636680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.637571 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.637574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.638588 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.638599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.639242 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.639245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.639308 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.639765 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.639777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.640166 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.641787 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.641790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.641792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.642055 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.642300 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.642305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.642382 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.642723 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.642726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.642848 (+ 122us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.642995 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.643119 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"spinlock_wait_cycles":27904,"thread_start_us":107,"threads_started":1}
I20260809 03:30:32.643914 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.636173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.15.222.125:41829)
0809 03:30:32.636269 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.636273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.636446 (+ 173us) server_negotiation.cc:407] Connection header received
0809 03:30:32.636558 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.636560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.636598 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.636677 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.637176 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.637853 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.638654 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.638946 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.639039 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.640340 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.640344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.640346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.640368 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.642046 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642760 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.642764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.642765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.642862 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.643144 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.643147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.643149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.643324 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.643386 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.643685 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.643781 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":1792}
I20260809 03:30:32.644207 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.636027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:58722)
0809 03:30:32.636345 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.636358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.636477 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.636716 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.636720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.636935 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.637060 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.637068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.637957 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.637961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.638559 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.638568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.639066 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.639070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.639161 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.639663 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.639679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.640215 (+ 536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.642152 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.642157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.642370 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.642953 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.642956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.642958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.643050 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.643460 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.643462 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.643539 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.643627 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.644044 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":39,"thread_start_us":100,"threads_started":1}
I20260809 03:30:32.643918 27251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:32.644444 27251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } }
I20260809 03:30:32.645447 27251 leader_election.cc:288] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcf35202e96c43009c3150248750bfad (127.15.222.126:41409)
I20260809 03:30:32.645874 26403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c379d4bee144e8babd3ea195e61c613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf35202e96c43009c3150248750bfad" is_pre_election: true
I20260809 03:30:32.646507 27165 leader_election.cc:302] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4c379d4bee144e8babd3ea195e61c613; no voters: dcf35202e96c43009c3150248750bfad
I20260809 03:30:32.646904 27251 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.126@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:32.656448 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.648831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.15.222.125:41829)
0809 03:30:32.648928 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.648932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.649231 (+ 299us) server_negotiation.cc:407] Connection header received
0809 03:30:32.649405 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.649408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.649453 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.649513 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.650138 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.650587 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.651395 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.651601 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.651681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.653628 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.653631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.653634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.653656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.654863 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.655341 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.655345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.655347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.655422 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.655660 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.655663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.655665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.655817 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.655905 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.656205 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.656306 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:30:32.656476 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.648807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:58738)
0809 03:30:32.649126 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.649163 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.649287 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.649559 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.649564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.649936 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.650034 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.650041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.650681 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.650685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.651288 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.651296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.651694 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.651697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.651807 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.652238 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.652249 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.653508 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.654968 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.654978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.654984 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.655217 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.655500 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.655503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.655506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.655584 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.655911 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.655918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.656027 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.656143 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.656315 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260809 03:30:32.657823 26387 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.222.125:41829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4c379d4bee144e8babd3ea195e61c613"
member_type: NON_VOTER
last_known_addr {
host: "127.15.222.125"
port: 41829
}
attrs {
promote: true
}
}
dest_uuid: "dcf35202e96c43009c3150248750bfad"
cas_config_opid_index: -1
I20260809 03:30:32.658495 26387 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } }
I20260809 03:30:32.660324 27283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4c379d4bee144e8babd3ea195e61c613 (127.15.222.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } }
I20260809 03:30:32.661029 27207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:32.662315 27207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 03:30:32.662326 27283 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.222.125:41829
I20260809 03:30:32.662777 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } }
I20260809 03:30:32.662880 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:32.663108 27137 master.cc:537] Master@127.15.222.125:41829 shutting down...
I20260809 03:30:32.663718 27284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:30:32.666154 26362 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 03:30:32.666970 26362 consensus_peers.cc:487] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad -> Peer 4c379d4bee144e8babd3ea195e61c613 (127.15.222.125:41829): Couldn't send request to peer 4c379d4bee144e8babd3ea195e61c613. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 03:30:32.667856 27137 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:32.668102 27137 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:32.668195 27137 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: stopping tablet replica
I20260809 03:30:32.672847 27137 master.cc:552] Master@127.15.222.125:41829 shutdown complete.
I20260809 03:30:32.672899 27137 master_runner.cc:305] Clearing existing system tablet
I20260809 03:30:32.673137 27137 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.673194 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.15.222.124@KRBTEST.COM; defaulting to no policy
I20260809 03:30:32.674463 27137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.124@KRBTEST.COM" created.
I20260809 03:30:32.676818 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.677048 27137 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.677155 27137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:32.677186 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "4c379d4bee144e8babd3ea195e61c613"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e2bf3590128812a71c5b105ab7109b1e"
server_key_iv: "f429562859f22118e75ce297106a4e39"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.679268 27137 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:32.681785 27137 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 03:30:32.681888 27137 log.cc:1196] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 03:30:32.682124 27137 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Deleting consensus metadata
I20260809 03:30:32.682271 27137 master_runner.cc:315] Copying system tablet from 127.15.222.126:41409
I20260809 03:30:32.683373 27137 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: tablet copy: Beginning tablet copy session from remote peer at address 127.15.222.126:41409
I20260809 03:30:32.692502 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.684149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.15.222.126:41409)
0809 03:30:32.684263 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.684267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.684561 (+ 294us) server_negotiation.cc:407] Connection header received
0809 03:30:32.684678 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.684681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.684732 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.684829 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.685431 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.686312 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.687511 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.687862 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.687969 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.689208 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.689211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.689214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.689241 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.690837 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.691289 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.691292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.691295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.691389 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.691639 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.691642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.691644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.691851 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.691942 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.692227 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.692350 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 03:30:32.692516 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.684055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44386)
0809 03:30:32.684461 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.684475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.684597 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.684950 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.684953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.685177 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.685301 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.685308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.686458 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.686461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.687382 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.687393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.687979 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.687983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.688075 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.688639 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.688653 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.689068 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.690955 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.690958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.690961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.691206 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.691469 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.691472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.691474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.691563 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.691946 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.691949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.692057 (+ 108us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.692169 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.692311 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":168,"threads_started":1}
I20260809 03:30:32.693316 26418 tablet_copy_service.cc:140] P dcf35202e96c43009c3150248750bfad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4c379d4bee144e8babd3ea195e61c613 ({username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:44386)
I20260809 03:30:32.693439 26418 tablet_copy_service.cc:161] P dcf35202e96c43009c3150248750bfad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4c379d4bee144e8babd3ea195e61c613 at {username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:44386: session id = 4c379d4bee144e8babd3ea195e61c613-00000000000000000000000000000000
I20260809 03:30:32.694902 26418 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 03:30:32.696504 27137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 03:30:32.698915 27137 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: tablet copy: Starting download of 0 data blocks...
I20260809 03:30:32.699080 27137 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: tablet copy: Starting download of 1 WAL segments...
I20260809 03:30:32.701998 27137 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 03:30:32.703605 26418 tablet_copy_service.cc:342] P dcf35202e96c43009c3150248750bfad: Request end of tablet copy session 4c379d4bee144e8babd3ea195e61c613-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:44386
I20260809 03:30:32.703685 26418 tablet_copy_service.cc:433] P dcf35202e96c43009c3150248750bfad: ending tablet copy session 4c379d4bee144e8babd3ea195e61c613-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4c379d4bee144e8babd3ea195e61c613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.124.keytab.
Entry for principal oryx/127.15.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.124.keytab.
I20260809 03:30:32.706853 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:39739
--webserver_interface=127.15.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.124.keytab
--principal=oryx/127.15.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:32.710480 27137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:32.711947 27304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.712268 27308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.712420 27137 server_base.cc:1034] running on GCE node
W20260809 03:30:32.712522 27306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.712769 27137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:32.713063 27137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:32.714192 27137 hybrid_clock.cc:648] HybridClock initialized: now 1786246232714178 us; error 28 us; skew 500 ppm
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:32.715767 27137 init.cc:377] Logged in from keytab as oryx/127.15.222.125@KRBTEST.COM (short username oryx)
I20260809 03:30:32.716220 27137 webserver.cc:460] Webserver started at http://127.15.222.125:34209/ using document root <none> and password file <none>
I20260809 03:30:32.716404 27137 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.716440 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:32.717880 27137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:32.718830 27321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.719087 27137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:32.719159 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "4c379d4bee144e8babd3ea195e61c613"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e2bf3590128812a71c5b105ab7109b1e"
server_key_iv: "f429562859f22118e75ce297106a4e39"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.719233 27137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.759521 27137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:32.761605 27137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:32.766723 27137 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.125:41829
I20260809 03:30:32.766752 27400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.125:41829 every 8 connection(s)
I20260809 03:30:32.767483 27137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
I20260809 03:30:32.769243 27401 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:32.770522 27401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Bootstrap starting.
I20260809 03:30:32.780726 27401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:32.781047 27401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Bootstrap complete.
I20260809 03:30:32.781661 27401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } }
I20260809 03:30:32.781821 27401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c379d4bee144e8babd3ea195e61c613, State: Initialized, Role: LEARNER
I20260809 03:30:32.781936 27401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } }
I20260809 03:30:32.782266 27405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } }
I20260809 03:30:32.782401 27401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:32.782447 27405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:30:32.785171 27421 catalog_manager.cc:1259] Loaded cluster ID: f4f4b7b1086a4ad0a66cc5856b5a7c7a
I20260809 03:30:32.785282 27421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: loading cluster ID for follower catalog manager: success
I20260809 03:30:32.787427 27421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: acquiring CA information for follower catalog manager: success
I20260809 03:30:32.788834 27421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 03:30:32.836859 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:32.837105 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:32.837141 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:32.841030 27302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:32.841087 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:32.841104 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:32.841118 27302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:32.841135 27302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:32.846808 27302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.124.keytab
--principal=oryx/127.15.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:32.848101 27302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:32.849404 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:32.858130 27436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.858168 27426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:32.858167 27427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:32.858425 27302 server_base.cc:1034] running on GCE node
I20260809 03:30:32.859045 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:32.860095 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:32.861238 27302 hybrid_clock.cc:648] HybridClock initialized: now 1786246232861214 us; error 35 us; skew 500 ppm
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:32.863873 27302 init.cc:377] Logged in from keytab as oryx/127.15.222.124@KRBTEST.COM (short username oryx)
I20260809 03:30:32.864694 27302 webserver.cc:460] Webserver started at http://127.15.222.124:35405/ using document root <none> and password file <none>
I20260809 03:30:32.865218 27302 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:32.865283 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:32.865471 27302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:32.867472 27302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/instance:
uuid: "344276192d594e56be9aeea2cf783269"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e52e6881ab8b796a75f26d70e3dcd8d3"
server_key_iv: "9c053cd2c4f3f751586647182ae5d365"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.868079 27302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal/instance:
uuid: "344276192d594e56be9aeea2cf783269"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e52e6881ab8b796a75f26d70e3dcd8d3"
server_key_iv: "9c053cd2c4f3f751586647182ae5d365"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.871627 27302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:32.873847 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:32.875087 27302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:30:32.875195 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "344276192d594e56be9aeea2cf783269"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e52e6881ab8b796a75f26d70e3dcd8d3"
server_key_iv: "9c053cd2c4f3f751586647182ae5d365"
server_key_version: "encryptionkey@0"
I20260809 03:30:32.875276 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:32.889374 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:32.892007 27302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:32.892149 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:32.898646 27302 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.124:39739
I20260809 03:30:32.898979 27529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.124:39739 every 8 connection(s)
I20260809 03:30:32.899575 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
I20260809 03:30:32.902261 27530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:32.904476 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 27302
I20260809 03:30:32.906960 27530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.124@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:32.917631 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.908878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.15.222.126:41409)
0809 03:30:32.909005 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.909009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.909141 (+ 132us) server_negotiation.cc:407] Connection header received
0809 03:30:32.909257 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.909260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.909322 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.909403 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.910121 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.910826 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.911834 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.912054 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.912137 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.914354 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.914357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.914359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.914393 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.915626 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.916115 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.916118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.916119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.916193 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.916521 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.916524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.916526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.916710 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.916799 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.917350 (+ 551us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.917467 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:30:32.918604 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.908688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44390)
0809 03:30:32.909020 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.909067 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.909180 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.909448 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.909453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.909866 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.910008 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.910014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.910943 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.910946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.911730 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.911738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.912182 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.912186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.912276 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.912711 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.912725 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.914215 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.915724 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.915733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.915743 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.916031 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.916265 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.916267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.916268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.916388 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.916812 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.916817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.917131 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.917286 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.917582 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260809 03:30:32.918608 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.905826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.15.222.124:39739)
0809 03:30:32.906187 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.906202 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.906236 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:32.906864 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.906880 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.907184 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.907489 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.908301 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.909618 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.910679 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.911255 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.911417 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.913821 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.913827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.913840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.913867 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.915701 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.916293 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.916298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.916299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.916423 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.917032 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.917034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.917035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.917246 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.917457 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.917691 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.917853 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260809 03:30:32.919834 27530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.124@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:32.927310 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.920247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:41390)
0809 03:30:32.920339 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.920352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.920467 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.920796 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.920798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.920931 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.921016 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.921021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.921889 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.921892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.922389 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.922397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.922750 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.922752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.922820 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.923150 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.923159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.924420 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.926094 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.926097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.926099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.926274 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.926521 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.926524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.926526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.926601 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.926964 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.926966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.927038 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.927111 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.927202 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 03:30:32.927359 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.920348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.15.222.125:41829)
0809 03:30:32.920609 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.920613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.920624 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:30:32.920658 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.920660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.920696 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.920762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.921098 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.921794 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.922471 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.922680 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.922747 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.924557 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.924560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.924563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.924591 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.925999 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.926358 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.926362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.926363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.926441 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.926712 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.926717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.926718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.926888 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.926968 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.927154 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.927245 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260809 03:30:32.927970 27530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246232, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.124@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
I20260809 03:30:32.934824 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.928402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.0.0.1:42252)
0809 03:30:32.928503 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.928516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.928578 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.928741 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.928742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.928854 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.928928 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.928933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.929939 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.929942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.930414 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.930421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.930854 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.930856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.930938 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.931288 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.931301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.932355 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.933570 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.933573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.933575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.933768 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.933999 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.934002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.934004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.934076 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.934372 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.934375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.934458 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.934537 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.934655 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":13}
I20260809 03:30:32.934824 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.928430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.15.222.124:39739)
0809 03:30:32.928531 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.928535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.928549 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:32.928636 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.928638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.928669 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.928724 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.929001 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.929454 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.930515 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.930776 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.930848 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.932472 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.932474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.932476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.932494 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.933482 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.933843 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.933847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.933849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.933924 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.934146 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.934149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.934151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.934292 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.934370 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.934590 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.934705 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":38}
I20260809 03:30:32.937575 27530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Bootstrap starting.
I20260809 03:30:32.939824 27530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:32.940523 27530 log.cc:824] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:32.942281 27530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: No bootstrap required, opened a new log
I20260809 03:30:32.945065 27530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } }
I20260809 03:30:32.945245 27530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:32.945295 27530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344276192d594e56be9aeea2cf783269, State: Initialized, Role: FOLLOWER
I20260809 03:30:32.945681 27530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } }
I20260809 03:30:32.946134 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } } }
I20260809 03:30:32.946245 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:32.946831 27530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 03:30:32.951181 27556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:32.951248 27556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:32.954254 27302 master_runner.cc:428] Detected that this master 344276192d594e56be9aeea2cf783269 is joining an existing cluster
I20260809 03:30:32.954310 27302 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.222.124:39739
I20260809 03:30:32.957789 26709 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
I20260809 03:30:32.964859 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.956594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.15.222.126:41409)
0809 03:30:32.956991 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.956995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.957010 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:32.957051 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.957054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.957100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.957174 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.958510 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.959300 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.960642 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.960974 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.961045 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.962222 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.962225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.962228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.962253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.963439 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.963804 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.963808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.963810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.963883 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.964123 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.964126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.964128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.964299 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:32.964383 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:32.964627 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 03:30:32.964728 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260809 03:30:32.964871 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.956548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44404)
0809 03:30:32.956843 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.956858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.956945 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.957390 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.957393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.958239 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.958362 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.958370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.959435 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.959437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.960510 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.960521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.961194 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.961197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.961282 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.961739 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.961754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.962098 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.963538 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.963541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.963543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.963721 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.963964 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.963967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.963969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.964048 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.964385 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:32.964389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:32.964477 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:30:32.964560 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:30:32.964687 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":145,"thread_start_us":117,"threads_started":1}
I20260809 03:30:32.971621 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 26485
Aug 09 03:30:32 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246231, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.126@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
I20260809 03:30:33.001173 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.957305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.15.222.124:39739)
0809 03:30:32.957396 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.957400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.957410 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:30:32.957438 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.957440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.957469 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.957528 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.958069 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.958733 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.959977 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.960310 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.960397 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.994237 (+ 33840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.994241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.994244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.994270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:32.998761 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:32.999329 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:32.999332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:32.999334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:32.999413 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.000197 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.000200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.000201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.000452 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.000534 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.000884 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.001011 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30,"spinlock_wait_cycles":1424128}
I20260809 03:30:33.001758 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.956584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.0.0.1:42258)
0809 03:30:32.957221 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.957234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.957312 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.957573 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.957576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.957814 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.957956 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.957964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.958858 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.958861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.959811 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.959822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.960577 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.960580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.960677 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.961156 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.961170 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.994017 (+ 32847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:32.998875 (+ 4858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.998878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.998880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.999181 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:32.999747 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:32.999750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:32.999752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:32.999851 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.000592 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.000594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.000675 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.000759 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.001573 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":6,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260809 03:30:33.003785 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:44941
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:41409
--webserver_port=41829
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:33.004124 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.957250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.15.222.125:41829)
0809 03:30:32.957358 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.957363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.957570 (+ 207us) server_negotiation.cc:407] Connection header received
0809 03:30:32.957750 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.957753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.957796 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.957844 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.958509 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.959275 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.960642 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.961284 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.961355 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.999173 (+ 37818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.999176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.999179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.999231 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.000827 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.002064 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.002067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.002069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.002156 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.002461 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.002464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.002466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.002677 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.002759 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.003826 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.003963 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:33.004693 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.956976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:41394)
0809 03:30:32.957494 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.957507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.957630 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.958040 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.958043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.958271 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.958374 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.958380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.959426 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.959429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.960510 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.960521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.961511 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.961513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.961579 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.997435 (+ 35856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.997453 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.998925 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.000956 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.000959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.000961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.001184 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.002261 (+ 1077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.002264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.002266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.002347 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.002829 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.002832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.002935 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.004300 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.004454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":92,"thread_start_us":83,"threads_started":1}
I20260809 03:30:33.006023 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.968094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.0.0.1:42270)
0809 03:30:32.968200 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:32.968215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:32.968293 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:32.968666 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:32.968668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:32.968801 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:32.968878 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.968883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.969863 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.969865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.970473 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:32.970482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.971061 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.971065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:32.971155 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.971590 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:32.971604 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:32.984617 (+ 13013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.003092 (+ 18475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.003096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.003098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.003399 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.003761 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.003765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.003767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.003870 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.005476 (+ 1606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.005480 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.005590 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.005672 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.005843 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:30:33.006642 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:32.968152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.15.222.124:39739)
0809 03:30:32.968434 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:32.968442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:32.968455 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:32.968516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:32.968518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:32.968565 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:32.968642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:32.968976 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.969780 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.970571 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:32.970910 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:32.971043 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:32.985910 (+ 14867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:32.985914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:32.985916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:32.985947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.001930 (+ 15983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.003518 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.003521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.003522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.003654 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.005042 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.005046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.005047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.005339 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.006286 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.006359 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.006485 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260809 03:30:33.007440 26387 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.222.124:39739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "344276192d594e56be9aeea2cf783269"
member_type: NON_VOTER
last_known_addr {
host: "127.15.222.124"
port: 39739
}
attrs {
promote: true
}
}
dest_uuid: "dcf35202e96c43009c3150248750bfad"
cas_config_opid_index: 5
I20260809 03:30:33.008145 26387 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } }
I20260809 03:30:33.009332 27285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 344276192d594e56be9aeea2cf783269 (127.15.222.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.009837 27492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:33.010866 27285 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.222.124:39739
I20260809 03:30:33.011053 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.011142 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.011204 27492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 03:30:33.011824 27284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:33.012251 27302 master.cc:537] Master@127.15.222.124:39739 shutting down...
W20260809 03:30:33.013808 26367 consensus_peers.cc:487] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad -> Peer 344276192d594e56be9aeea2cf783269 (127.15.222.124:39739): Couldn't send request to peer 344276192d594e56be9aeea2cf783269. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 03:30:33.015935 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.009464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:41396)
0809 03:30:33.009988 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.010001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.010081 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.010516 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.010519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.010537 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.010624 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.010630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.012817 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.012822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.014176 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.014188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.015526 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.015530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.015623 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.015672 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.015742 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.015813 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448}
I20260809 03:30:33.016350 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.009673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.15.222.125:41829)
0809 03:30:33.009889 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.009893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.010195 (+ 302us) server_negotiation.cc:407] Connection header received
0809 03:30:33.010239 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.010242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.010296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.010395 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.010735 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.011578 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.014318 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.014981 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.015090 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.015236 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.016164 (+ 928us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.016245 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260809 03:30:33.017427 27364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 LEARNER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 03:30:33.018067 27284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 03:30:33.019228 27302 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:33.019579 27302 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:33.019691 27302 tablet_replica.cc:330] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: stopping tablet replica
I20260809 03:30:33.019928 27364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 344276192d594e56be9aeea2cf783269 (127.15.222.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.022105 27405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf35202e96c43009c3150248750bfad. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.022233 27405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:30:33.022315 27405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: NON_VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: true } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.022385 27405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:30:33.023775 27284 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: attempting to promote NON_VOTER 4c379d4bee144e8babd3ea195e61c613 to VOTER
I20260809 03:30:33.024226 27284 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } }
I20260809 03:30:33.025359 27364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 LEARNER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 03:30:33.026098 27285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 03:30:33.026881 26367 consensus_peers.cc:487] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad -> Peer 344276192d594e56be9aeea2cf783269 (127.15.222.124:39739): Couldn't send request to peer 344276192d594e56be9aeea2cf783269. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 03:30:33.027758 27285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4c379d4bee144e8babd3ea195e61c613 (127.15.222.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.029129 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.029224 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.029348 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.029420 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.029958 27364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4c379d4bee144e8babd3ea195e61c613 (127.15.222.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.031512 27578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.031652 27578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:33.039292 27302 master.cc:552] Master@127.15.222.124:39739 shutdown complete.
I20260809 03:30:33.039345 27302 master_runner.cc:305] Clearing existing system tablet
I20260809 03:30:33.039583 27302 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:33.039637 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:33.040902 27302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 03:30:33.041882 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.042097 27302 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:33.042227 27302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:33.042266 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "344276192d594e56be9aeea2cf783269"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e52e6881ab8b796a75f26d70e3dcd8d3"
server_key_iv: "9c053cd2c4f3f751586647182ae5d365"
server_key_version: "encryptionkey@0"
I20260809 03:30:33.044281 27302 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:33.046603 27302 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 03:30:33.046691 27302 log.cc:1196] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 03:30:33.046940 27302 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Deleting consensus metadata
I20260809 03:30:33.047097 27302 master_runner.cc:315] Copying system tablet from 127.15.222.126:41409
I20260809 03:30:33.048199 27302 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: tablet copy: Beginning tablet copy session from remote peer at address 127.15.222.126:41409
I20260809 03:30:33.057173 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.048796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.15.222.126:41409)
0809 03:30:33.048915 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.048920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.049191 (+ 271us) server_negotiation.cc:407] Connection header received
0809 03:30:33.049328 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.049331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.049389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.049478 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.050074 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.050900 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.052024 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.052356 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.052463 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.053733 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.053737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.053740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.053767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.055541 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.055991 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.055994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.055996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.056083 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.056336 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.056338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.056340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.056552 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.056644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.056900 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.057014 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:33.057188 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.048795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44416)
0809 03:30:33.049092 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.049106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.049237 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.049524 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.049527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.049798 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.049955 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.049963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.051017 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.051020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.051904 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.051916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.052481 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.052484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.052581 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.053126 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.053141 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.053568 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.055636 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.055640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.055642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.055883 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.056163 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.056166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.056167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.056256 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.056638 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.056641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.056741 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.056844 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.056987 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260809 03:30:33.058015 26418 tablet_copy_service.cc:140] P dcf35202e96c43009c3150248750bfad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 344276192d594e56be9aeea2cf783269 ({username='oryx', principal='oryx/127.15.222.124@KRBTEST.COM'} at 127.0.0.1:44416)
I20260809 03:30:33.058117 26418 tablet_copy_service.cc:161] P dcf35202e96c43009c3150248750bfad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 344276192d594e56be9aeea2cf783269 at {username='oryx', principal='oryx/127.15.222.124@KRBTEST.COM'} at 127.0.0.1:44416: session id = 344276192d594e56be9aeea2cf783269-00000000000000000000000000000000
I20260809 03:30:33.059211 26418 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 03:30:33.060556 27302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 03:30:33.062937 27302 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: tablet copy: Starting download of 0 data blocks...
I20260809 03:30:33.063124 27302 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: tablet copy: Starting download of 1 WAL segments...
I20260809 03:30:33.064549 27302 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 03:30:33.066267 26418 tablet_copy_service.cc:342] P dcf35202e96c43009c3150248750bfad: Request end of tablet copy session 344276192d594e56be9aeea2cf783269-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.222.124@KRBTEST.COM'} at 127.0.0.1:44416
I20260809 03:30:33.066352 26418 tablet_copy_service.cc:433] P dcf35202e96c43009c3150248750bfad: ending tablet copy session 344276192d594e56be9aeea2cf783269-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 344276192d594e56be9aeea2cf783269
I20260809 03:30:33.073958 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:33.075577 27593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.075810 27594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.076248 27596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.076752 27302 server_base.cc:1034] running on GCE node
I20260809 03:30:33.076942 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:33.077276 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:33.078449 27302 hybrid_clock.cc:648] HybridClock initialized: now 1786246233078433 us; error 32 us; skew 500 ppm
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:33.080220 27302 init.cc:377] Logged in from keytab as oryx/127.15.222.124@KRBTEST.COM (short username oryx)
I20260809 03:30:33.080731 27302 webserver.cc:460] Webserver started at http://127.15.222.124:41859/ using document root <none> and password file <none>
I20260809 03:30:33.080978 27302 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:33.081066 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:33.082574 27302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:33.083813 27609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.084048 27302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:33.084158 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "344276192d594e56be9aeea2cf783269"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "e52e6881ab8b796a75f26d70e3dcd8d3"
server_key_iv: "9c053cd2c4f3f751586647182ae5d365"
server_key_version: "encryptionkey@0"
I20260809 03:30:33.084300 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:33.120177 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:33.122474 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:33.127522 27302 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.124:39739
I20260809 03:30:33.128024 27689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.124:39739 every 8 connection(s)
I20260809 03:30:33.128293 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
I20260809 03:30:33.129702 27690 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:33.130539 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Bootstrap starting.
I20260809 03:30:33.138878 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:33.139114 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: Bootstrap complete.
I20260809 03:30:33.139535 27690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } }
I20260809 03:30:33.139647 27690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344276192d594e56be9aeea2cf783269, State: Initialized, Role: LEARNER
I20260809 03:30:33.139744 27690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } }
I20260809 03:30:33.140043 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.140158 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:30:33.140184 27690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:33.142760 27722 catalog_manager.cc:1259] Loaded cluster ID: f4f4b7b1086a4ad0a66cc5856b5a7c7a
I20260809 03:30:33.144924 27722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: loading cluster ID for follower catalog manager: success
I20260809 03:30:33.147080 27722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: acquiring CA information for follower catalog manager: success
I20260809 03:30:33.148316 27722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 03:30:33.148881 27574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:33.149091 27574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:33.149133 27574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:33.153034 27574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:33.153083 27574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:33.153139 27574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:33.158124 27574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:44941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=41829
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:33.159067 27574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:33.160058 27574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:33.162274 26917 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
W20260809 03:30:33.169230 27733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.169217 27736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.169328 27574 server_base.cc:1034] running on GCE node
W20260809 03:30:33.169219 27738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.169872 27574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:33.170449 27574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:33.171602 27574 hybrid_clock.cc:648] HybridClock initialized: now 1786246233171573 us; error 44 us; skew 500 ppm
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:33.174211 27574 init.cc:377] Logged in from keytab as oryx/127.15.222.65@KRBTEST.COM (short username oryx)
I20260809 03:30:33.174997 27574 webserver.cc:460] Webserver started at http://127.15.222.65:41829/ using document root <none> and password file <none>
I20260809 03:30:33.175513 27574 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:33.175597 27574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:33.179215 27574 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:33.181099 27748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.182359 27574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:33.182462 27574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "001906698b2441979f3f786812886e37"
format_stamp: "Formatted at 2026-08-09 03:30:31 on dist-test-slave-cvwc"
server_key: "7556661841d757723c5b64b631ee6b0d"
server_key_iv: "2f2b179c30b36897a4c83d0fb92c694e"
server_key_version: "encryptionkey@0"
I20260809 03:30:33.183069 27574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:33.194917 27574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:33.197369 27574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:33.197573 27574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:33.198145 27574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:33.199009 27574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:33.199057 27574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.199107 27574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:33.199133 27574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.207371 27574 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44941
I20260809 03:30:33.207397 27927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44941 every 8 connection(s)
I20260809 03:30:33.208285 27574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:33.211853 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 27574
I20260809 03:30:33.212018 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 26716
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
I20260809 03:30:33.230059 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.210008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:38547 (local address 127.15.222.126:41409)
0809 03:30:33.210145 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.210149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.211201 (+ 1052us) server_negotiation.cc:407] Connection header received
0809 03:30:33.218068 (+ 6867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.218073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.218146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.218240 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.219968 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.220631 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.222021 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.222271 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.222339 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.225103 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.225107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.225109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.225133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.228095 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.228588 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.228591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.228593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.228664 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.228982 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.228985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.228986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.229162 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.229226 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.229807 (+ 581us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.229918 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 03:30:33.231307 27933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.210528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:38547)
0809 03:30:33.211043 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.211099 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.217618 (+ 6519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.218392 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.218395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.219066 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.219816 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.219831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.220794 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.220797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.221899 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.221910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.222478 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.222481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.222677 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.223211 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.223228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.224944 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.228215 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.228220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.228233 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.228497 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.228737 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.228739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.228741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.228895 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.229267 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.229272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.229515 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.229668 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.230468 (+ 800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":31,"spinlock_wait_cycles":297216,"thread_start_us":141,"threads_started":1}
I20260809 03:30:33.232997 27929 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:33.233210 27929 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.233662 27929 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:33.234738 26387 ts_manager.cc:194] Re-registered known tserver with Master: 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
I20260809 03:30:33.235353 26387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:38547
I20260809 03:30:33.236181 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.210080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:33783 (local address 127.15.222.124:39739)
0809 03:30:33.210291 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.210294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.211201 (+ 907us) server_negotiation.cc:407] Connection header received
0809 03:30:33.218590 (+ 7389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.218593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.218647 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.218698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.220035 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.221476 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.223068 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.223530 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.223620 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.231761 (+ 8141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.231766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.231769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.231803 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.233623 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.234335 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.234340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.234342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.234454 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.234941 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.234945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.234948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.235221 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.235320 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.235818 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.235994 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260809 03:30:33.236976 27934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.210528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.15.222.65:33783)
0809 03:30:33.211042 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.211099 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.217614 (+ 6515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.218815 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.218817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.219324 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.219821 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.219835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.221685 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.221689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.222887 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.222912 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.223804 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.223808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.223942 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.224393 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.224408 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.231329 (+ 6921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.233883 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.233887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.233889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.234193 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.234592 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.234595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.234598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.234709 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.235392 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.235395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.235579 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.236566 (+ 987us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.236731 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":26,"spinlock_wait_cycles":226560,"thread_start_us":121,"threads_started":1}
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:33.243202 27928 heartbeater.cc:347] Connected to a master server at 127.15.222.124:39739
I20260809 03:30:33.243295 27928 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.243489 27928 heartbeater.cc:511] Master 127.15.222.124:39739 requested a full tablet report, sending...
I20260809 03:30:33.244685 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.211707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:35413 (local address 127.15.222.125:41829)
0809 03:30:33.211831 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.211834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.212079 (+ 245us) server_negotiation.cc:407] Connection header received
0809 03:30:33.217931 (+ 5852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.217934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.217987 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.218029 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.235912 (+ 17883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.236529 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.237408 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.237887 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.238028 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.241493 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.241497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.241500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.241526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.243098 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.243562 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.243566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.243567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.243658 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.243932 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.243935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.243937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.244113 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.244198 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.244435 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.244547 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 03:30:33.244654 27635 ts_manager.cc:194] Registered new tserver with Master: 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
I20260809 03:30:33.244700 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.211564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.15.222.65:35413)
0809 03:30:33.211987 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.211999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.217818 (+ 5819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.218314 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.218321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.219082 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.232891 (+ 13809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.232899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.236651 (+ 3752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.236653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.237304 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.237311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.238205 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.238210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.238321 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.238748 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.238764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.241341 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.243217 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.243220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.243222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.243458 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.243747 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.243750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.243752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.243844 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.244202 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.244205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.244288 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.244372 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.244530 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":94,"threads_started":1}
I20260809 03:30:33.245306 27930 heartbeater.cc:347] Connected to a master server at 127.15.222.125:41829
I20260809 03:30:33.245389 27930 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.245513 27930 heartbeater.cc:511] Master 127.15.222.125:41829 requested a full tablet report, sending...
I20260809 03:30:33.246577 27338 ts_manager.cc:194] Registered new tserver with Master: 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
I20260809 03:30:33.248193 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:42195
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:41409
--webserver_port=39983
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:33.357649 27937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:33.357928 27937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:33.357987 27937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:33.361917 27937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:33.361965 27937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:33.362039 27937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:33.367297 27937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:42195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=39983
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:33.367588 27128 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
I20260809 03:30:33.368257 27937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:33.369215 27937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:33.378301 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.378319 27947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.378307 27948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.378407 27937 server_base.cc:1034] running on GCE node
I20260809 03:30:33.378924 27937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:33.379501 27937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:33.380661 27937 hybrid_clock.cc:648] HybridClock initialized: now 1786246233380644 us; error 29 us; skew 500 ppm
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:33.383423 27937 init.cc:377] Logged in from keytab as oryx/127.15.222.66@KRBTEST.COM (short username oryx)
I20260809 03:30:33.384361 27937 webserver.cc:460] Webserver started at http://127.15.222.66:39983/ using document root <none> and password file <none>
I20260809 03:30:33.384963 27937 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:33.385049 27937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:33.388742 27937 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:33.390712 27960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.391893 27937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:33.391996 27937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "2d864ffd4065481ead0362cc6fc23f3d"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "d33f94e51e1aa677d283c4e3203c3542"
server_key_iv: "f021b8306d2b57d797d1a459fe8d4754"
server_key_version: "encryptionkey@0"
I20260809 03:30:33.392517 27937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:33.402626 27937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:33.405407 27937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:33.405570 27937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:33.406131 27937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:33.406929 27937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:33.406976 27937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.407027 27937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:33.407054 27937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.415006 27937 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42195
I20260809 03:30:33.415059 28143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42195 every 8 connection(s)
I20260809 03:30:33.415932 27937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:33.423804 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 27937
I20260809 03:30:33.423977 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 26924
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:33.430845 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.419091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34279 (local address 127.15.222.125:41829)
0809 03:30:33.419212 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.419217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.419463 (+ 246us) server_negotiation.cc:407] Connection header received
0809 03:30:33.420098 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.420101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.420152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.420223 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.421310 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.421956 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423556 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.423887 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423984 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.426910 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.426914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.426917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.426946 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.428484 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429105 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.429109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.429111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.429197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429593 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.429596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.429598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.429850 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.429933 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.430570 (+ 637us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.430688 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:33.431226 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.417659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:53047 (local address 127.15.222.124:39739)
0809 03:30:33.417802 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.417806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.418611 (+ 805us) server_negotiation.cc:407] Connection header received
0809 03:30:33.419850 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.419853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.419921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.419992 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.421301 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.422172 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423287 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.423544 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423636 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.427302 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.427305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.427308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.427337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.429430 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429961 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.429965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.429966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.430052 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.430348 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.430352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.430353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.430572 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.430644 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.430965 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.431089 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 03:30:33.432310 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.418972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.15.222.66:34279)
0809 03:30:33.419367 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.419379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.419619 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.420348 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.420351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.420599 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.421157 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.421171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.422132 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.422135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.423442 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.423451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.424103 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.424106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.424187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.424643 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.424656 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.426727 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.428609 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.428615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.428629 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.428987 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.429310 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.429315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.429492 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.430010 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.430016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.430299 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.431015 (+ 716us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.431315 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":181,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260809 03:30:33.432338 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.417620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46771 (local address 127.15.222.126:41409)
0809 03:30:33.417800 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.417804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.418784 (+ 980us) server_negotiation.cc:407] Connection header received
0809 03:30:33.419850 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.419853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.419909 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.419963 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.421321 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.422145 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423573 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.423865 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423952 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.427604 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.427608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.427611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.427641 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.429478 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.430127 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.430131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.430134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.430241 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.430640 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.430645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.430647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.430915 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.431005 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.432018 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.432161 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 03:30:33.432582 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.418009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.15.222.66:53047)
0809 03:30:33.418450 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.418512 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.419615 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.420146 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.420153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.420595 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.421155 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.421168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.422293 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.422295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.423173 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.423182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.423791 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.423795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.423985 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.424534 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.424556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.427136 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.429539 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.429543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.429850 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.430156 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.430158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.430160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.430245 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.430703 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.430705 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.430787 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.431348 (+ 561us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.431472 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":170,"spinlock_wait_cycles":30848,"thread_start_us":100,"threads_started":1}
I20260809 03:30:33.433468 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.418009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:46771)
0809 03:30:33.418696 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.418710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.419620 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.420147 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.420155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.420605 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.421157 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.421171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.422248 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.422251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.423440 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.423451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.424197 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.424201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.424757 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.425230 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.425246 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.427432 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.429620 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.429624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.429627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.429994 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.430391 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.430395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.430397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.430507 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.431647 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.431651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.431766 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.433092 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.433255 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":250,"spinlock_wait_cycles":27904,"thread_start_us":156,"threads_started":1}
I20260809 03:30:33.433511 28144 heartbeater.cc:347] Connected to a master server at 127.15.222.124:39739
I20260809 03:30:33.433857 28144 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.434327 28144 heartbeater.cc:511] Master 127.15.222.124:39739 requested a full tablet report, sending...
I20260809 03:30:33.435353 27635 ts_manager.cc:194] Registered new tserver with Master: 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:33.442781 28146 heartbeater.cc:347] Connected to a master server at 127.15.222.125:41829
I20260809 03:30:33.442867 28146 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.443025 28146 heartbeater.cc:511] Master 127.15.222.125:41829 requested a full tablet report, sending...
I20260809 03:30:33.443780 27338 ts_manager.cc:194] Registered new tserver with Master: 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:33.443955 28145 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:33.444034 28145 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.444190 28145 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:33.444757 26387 ts_manager.cc:194] Re-registered known tserver with Master: 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:33.445304 26387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:46771
I20260809 03:30:33.449507 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:37765
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:41409
--webserver_port=33339
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:33.559163 28155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:33.559391 28155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:33.559433 28155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:33.563340 28155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:33.563391 28155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:33.563458 28155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:33.568481 28155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:37765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=33339
--tserver_master_addrs=127.15.222.126:41409,127.15.222.125:41829,127.15.222.124:39739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:33.569476 28155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:33.570528 28155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:33.580107 28167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.580225 28155 server_base.cc:1034] running on GCE node
W20260809 03:30:33.580098 28165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:33.580093 28170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:33.580826 28155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:33.581434 28155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:33.582589 28155 hybrid_clock.cc:648] HybridClock initialized: now 1786246233582560 us; error 41 us; skew 500 ppm
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:33.585479 28155 init.cc:377] Logged in from keytab as oryx/127.15.222.67@KRBTEST.COM (short username oryx)
I20260809 03:30:33.586388 28155 webserver.cc:460] Webserver started at http://127.15.222.67:33339/ using document root <none> and password file <none>
I20260809 03:30:33.586920 28155 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:33.587004 28155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:33.590646 28155 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:33.592545 28178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.593644 28155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:33.593765 28155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "557595f6ed2d40b9840fd838e7feadf9"
format_stamp: "Formatted at 2026-08-09 03:30:32 on dist-test-slave-cvwc"
server_key: "388cc52336606ed952466f7e55dbae3b"
server_key_iv: "ec0e07733ba4d4d34224208324debb74"
server_key_version: "encryptionkey@0"
I20260809 03:30:33.594285 28155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:33.616513 28155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:33.619794 28155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:33.619975 28155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:33.620652 28155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:33.621438 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.617566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.0.0.1:42272)
0809 03:30:33.617848 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.617860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.617983 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.618257 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.618260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.618279 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.618387 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.618393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.619286 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.619289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.620182 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.620193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.621089 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.621092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.621166 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.621189 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.621237 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.621300 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260809 03:30:33.621487 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.617633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.15.222.124:39739)
0809 03:30:33.617772 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.617776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.617939 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:30:33.618096 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.618099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.618151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.618228 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.618501 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.619170 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.620317 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.620970 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.621102 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.621248 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.621288 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.621365 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:33.621752 28155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:33.621810 28155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.621873 28155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:33.621902 28155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:33.622208 27644 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 03:30:33.622578 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf35202e96c43009c3150248750bfad. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.622733 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: LEARNER
I20260809 03:30:33.630502 28155 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:37765
I20260809 03:30:33.630527 28370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:37765 every 8 connection(s)
I20260809 03:30:33.631459 28155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:33.635355 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 28155
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.124@KRBTEST.COM
Aug 09 03:30:33 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246233, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.125@KRBTEST.COM
I20260809 03:30:33.650449 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.633153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:57951 (local address 127.15.222.126:41409)
0809 03:30:33.633259 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.633262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.634610 (+ 1348us) server_negotiation.cc:407] Connection header received
0809 03:30:33.636260 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.636264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.636320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.636374 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.638356 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.639163 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641018 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.641352 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.644455 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.644458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.644461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.644489 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.646666 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.648572 (+ 1906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.648576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.648578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.648670 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.649162 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.649165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.649167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.649368 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.649446 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.650171 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.650282 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:30:33.650449 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.635262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:36487 (local address 127.15.222.125:41829)
0809 03:30:33.635377 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.635381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.635553 (+ 172us) server_negotiation.cc:407] Connection header received
0809 03:30:33.636251 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.636255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.636309 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.636359 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.641807 (+ 5448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.642469 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.643506 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.643827 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.643912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.646126 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.646130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.646132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.646158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.648074 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.648572 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.648576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.648578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.648773 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.649162 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.649165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.649167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.649368 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.649446 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.650168 (+ 722us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.650283 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:33.652000 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.15.222.67:36487)
0809 03:30:33.635488 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.635500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.636021 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.636751 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.636760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.639327 (+ 2567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.641627 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.641633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.642608 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.642611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.643367 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.643378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.644123 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.644127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.644217 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.644597 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.644611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.645974 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.648190 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.648193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.648195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.648477 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.648855 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.648858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.648860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.649052 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.649513 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.649519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.649889 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.650634 (+ 745us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.650961 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":1926,"spinlock_wait_cycles":9984,"thread_start_us":86,"threads_started":1}
I20260809 03:30:33.652482 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.633549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:57951)
0809 03:30:33.634471 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.634512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.636018 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.636751 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.636760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.637297 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.638195 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.638215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.639308 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.639311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.640878 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.640889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.641480 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.641672 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.642379 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.642406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.644249 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.646793 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.646799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.648061 (+ 1262us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.648460 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.648873 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.648876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.648877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.649052 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.649513 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.649519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.649889 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.650934 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.651042 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":210,"spinlock_wait_cycles":19456,"thread_start_us":610,"threads_started":1}
I20260809 03:30:33.653158 28376 heartbeater.cc:347] Connected to a master server at 127.15.222.125:41829
I20260809 03:30:33.653395 28376 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.653497 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.637541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.15.222.126:41409)
0809 03:30:33.637768 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.637772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.638786 (+ 1014us) server_negotiation.cc:407] Connection header received
0809 03:30:33.638902 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.638905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.638950 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.639030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.640113 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.640880 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.644491 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.644842 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.644919 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.647249 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.647253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.647255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.647280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.651108 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.652179 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.652183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.652184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.652266 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.652554 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.652557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.652559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.652758 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.652866 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.653212 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.653331 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260809 03:30:33.653995 28376 heartbeater.cc:511] Master 127.15.222.125:41829 requested a full tablet report, sending...
I20260809 03:30:33.654137 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.634974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:55341 (local address 127.15.222.124:39739)
0809 03:30:33.635087 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.635091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.635342 (+ 251us) server_negotiation.cc:407] Connection header received
0809 03:30:33.636252 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.636256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.636309 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.636359 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.638356 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.639161 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641018 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.641375 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641460 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.646915 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.646936 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.646938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.646964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.648678 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.652355 (+ 3677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.652358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.652360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.652448 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.653171 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.653175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.653177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.653380 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.653474 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.653886 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.653999 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":49}
I20260809 03:30:33.654605 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.634858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.15.222.67:55341)
0809 03:30:33.635253 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.635266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.636021 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.636751 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.636760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.637297 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:33.638195 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.638215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.639311 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.639313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.640894 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.640901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.641934 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.641937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.642354 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.643566 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:33.643581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:33.646752 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:33.650121 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.650124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.650126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.651947 (+ 1821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.652562 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:33.652565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:33.652566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:33.652642 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:33.653529 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:33.653532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:33.653620 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.654302 (+ 682us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.654449 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":160,"spinlock_wait_cycles":28416,"thread_start_us":116,"threads_started":1}
I20260809 03:30:33.654623 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.637966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.15.222.125:41829)
0809 03:30:33.638189 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.638193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.639141 (+ 948us) server_negotiation.cc:407] Connection header received
0809 03:30:33.640519 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.640522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.640571 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.640630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.642499 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.643276 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.645372 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.645760 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.645831 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.648811 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.648814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.648817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.648843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.651573 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.652754 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.652757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.652759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.652840 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.653791 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.653795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.653796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.653965 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.654083 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.654341 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.654468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260809 03:30:33.655287 28371 heartbeater.cc:347] Connected to a master server at 127.15.222.124:39739
I20260809 03:30:33.655320 27338 ts_manager.cc:194] Registered new tserver with Master: 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765)
I20260809 03:30:33.655365 28371 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.655611 28371 heartbeater.cc:511] Master 127.15.222.124:39739 requested a full tablet report, sending...
I20260809 03:30:33.656471 27635 ts_manager.cc:194] Registered new tserver with Master: 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765)
I20260809 03:30:33.656868 28374 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41409
I20260809 03:30:33.656798 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.638416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.15.222.124:39739)
0809 03:30:33.638657 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.638670 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.639433 (+ 763us) server_negotiation.cc:407] Connection header received
0809 03:30:33.639473 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.639476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.639520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.639598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.641332 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.643277 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.645961 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.646286 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.646382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.649618 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.649622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.649625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.649652 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.651483 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.655270 (+ 3787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.655273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.655275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.655363 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.655811 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.655814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.655816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.656027 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.656142 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.656418 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.656648 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260809 03:30:33.657150 28374 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:33.657374 28374 heartbeater.cc:511] Master 127.15.222.126:41409 requested a full tablet report, sending...
I20260809 03:30:33.658041 26385 ts_manager.cc:194] Re-registered known tserver with Master: 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765)
I20260809 03:30:33.658599 26385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:57951
I20260809 03:30:33.658021 26387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:33.661433 26387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:33.663792 28188 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: NON_VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: true } } }
I20260809 03:30:33.663928 28188 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.667603 28188 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: attempting to promote NON_VOTER 344276192d594e56be9aeea2cf783269 to VOTER
I20260809 03:30:33.667958 28188 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } }
I20260809 03:30:33.668973 27644 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 LEARNER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 03:30:33.668988 27364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Refusing update from remote peer dcf35202e96c43009c3150248750bfad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 03:30:33.669342 28404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 03:30:33.669472 28405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 03:30:33.671059 28402 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 344276192d594e56be9aeea2cf783269 (127.15.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.671576 27644 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 344276192d594e56be9aeea2cf783269 (127.15.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.672613 27364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 344276192d594e56be9aeea2cf783269 (127.15.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.673085 28402 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.673172 28402 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.673198 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.673285 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:33.673265 28402 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.673321 28402 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.673408 28402 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.673477 28402 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:33.674589 28414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf35202e96c43009c3150248750bfad" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:33.674693 28414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:33.682719 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.677275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44941 (local address 127.0.0.1:34972)
0809 03:30:33.677405 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.677418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.677482 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.679243 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.679246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.679267 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.679367 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.679372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.680515 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.680518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.681309 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.681319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.682317 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.682320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.682421 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.682449 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.682495 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.682585 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":17}
I20260809 03:30:33.683535 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.677470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.15.222.65:44941)
0809 03:30:33.677841 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.677854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.677878 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:30:33.677971 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.677978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.678206 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.679279 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.679485 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.680377 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.681456 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.682184 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.683094 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.683248 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.683299 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.683389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":138,"threads_started":1}
I20260809 03:30:33.685359 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.677906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42195 (local address 127.0.0.1:37256)
0809 03:30:33.679692 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.679705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.679792 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.680305 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.680308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.680327 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.680422 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.680428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.681773 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.681776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.683294 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.683304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.684988 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.684991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.685076 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.685097 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.685172 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.685237 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":101,"threads_started":1}
I20260809 03:30:33.685420 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.678071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.15.222.66:42195)
0809 03:30:33.678605 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.678618 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.679877 (+ 1259us) server_negotiation.cc:407] Connection header received
0809 03:30:33.679969 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.679974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.680164 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.680302 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.680519 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.681650 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.683423 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.684864 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.684974 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.685111 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.685200 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.685292 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":92,"threads_started":1}
I20260809 03:30:33.686728 27824 tablet_service.cc:1467] Processing CreateTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d (DEFAULT_TABLE table=test-table [id=5fefbd67ea8d4e54a0db35e6c725ecd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:33.687664 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.677912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37765 (local address 127.0.0.1:58988)
0809 03:30:33.682866 (+ 4954us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.682879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.682950 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.683564 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.683567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.683583 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.683651 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.683656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.685220 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.685224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.686315 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.686326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.687322 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.687326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.687408 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.687426 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.687483 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.687545 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4904,"thread_start_us":107,"threads_started":1}
I20260809 03:30:33.687736 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.678406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.15.222.67:37765)
0809 03:30:33.679567 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.679578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.683092 (+ 3514us) server_negotiation.cc:407] Connection header received
0809 03:30:33.683174 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.683179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.683408 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.683924 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.684059 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.685095 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.686455 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.687180 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.687287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.687419 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.687510 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.687595 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":114,"threads_started":1}
I20260809 03:30:33.687928 27824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9a9de3e4c14c3d916b604c4fac077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:33.690661 28263 tablet_service.cc:1467] Processing CreateTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d (DEFAULT_TABLE table=test-table [id=5fefbd67ea8d4e54a0db35e6c725ecd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:33.691680 28263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9a9de3e4c14c3d916b604c4fac077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:33.692126 28033 tablet_service.cc:1467] Processing CreateTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d (DEFAULT_TABLE table=test-table [id=5fefbd67ea8d4e54a0db35e6c725ecd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:33.692843 28033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9a9de3e4c14c3d916b604c4fac077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:33.699813 28425 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Bootstrap starting.
I20260809 03:30:33.700706 28424 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Bootstrap starting.
I20260809 03:30:33.701211 28426 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Bootstrap starting.
I20260809 03:30:33.702155 28425 tablet_bootstrap.cc:654] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:33.702890 28426 tablet_bootstrap.cc:654] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:33.702927 28424 tablet_bootstrap.cc:654] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:33.702998 28425 log.cc:824] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:33.703678 28426 log.cc:824] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:33.703833 28424 log.cc:824] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:33.705037 28425 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: No bootstrap required, opened a new log
I20260809 03:30:33.705226 28425 ts_tablet_manager.cc:1397] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:30:33.705510 28424 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: No bootstrap required, opened a new log
I20260809 03:30:33.705595 28426 tablet_bootstrap.cc:492] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: No bootstrap required, opened a new log
I20260809 03:30:33.705677 28424 ts_tablet_manager.cc:1397] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:33.705802 28426 ts_tablet_manager.cc:1397] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:33.708999 28424 raft_consensus.cc:348] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.709209 28424 raft_consensus.cc:374] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:33.709262 28424 raft_consensus.cc:729] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001906698b2441979f3f786812886e37, State: Initialized, Role: FOLLOWER
I20260809 03:30:33.709692 28424 consensus_queue.cc:260] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.710121 28426 raft_consensus.cc:348] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.710135 28425 raft_consensus.cc:348] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.710364 28426 raft_consensus.cc:374] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:33.710364 28425 raft_consensus.cc:374] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:33.710412 28426 raft_consensus.cc:729] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d864ffd4065481ead0362cc6fc23f3d, State: Initialized, Role: FOLLOWER
I20260809 03:30:33.710433 28425 raft_consensus.cc:729] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557595f6ed2d40b9840fd838e7feadf9, State: Initialized, Role: FOLLOWER
I20260809 03:30:33.710763 28424 ts_tablet_manager.cc:1428] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:30:33.710826 28425 consensus_queue.cc:260] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.710961 27929 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
I20260809 03:30:33.711596 28374 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
I20260809 03:30:33.711799 28425 ts_tablet_manager.cc:1428] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 03:30:33.710814 28426 consensus_queue.cc:260] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.716794 28426 ts_tablet_manager.cc:1428] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260809 03:30:33.717121 28145 heartbeater.cc:503] Master 127.15.222.126:41409 was elected leader, sending a full tablet report...
W20260809 03:30:33.775573 27392 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.15.222.125:41829
I20260809 03:30:33.818176 28430 raft_consensus.cc:482] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:33.818389 28430 raft_consensus.cc:504] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.819501 28430 leader_election.cc:288] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765), 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:33.824375 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.819892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42453 (local address 127.15.222.67:37765)
0809 03:30:33.820017 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.820021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.820217 (+ 196us) server_negotiation.cc:407] Connection header received
0809 03:30:33.820329 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.820332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.820380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.820469 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.820777 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.821562 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.823306 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.823870 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.823990 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.824034 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.824183 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.824256 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:33.824373 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.819722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37765 (local address 127.15.222.65:42453)
0809 03:30:33.820146 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.820157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.820235 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.820487 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.820490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.820513 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.820673 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.820681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.821656 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.821660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.823029 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.823041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.823986 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.823989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.824068 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.824091 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.824150 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.824207 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":130,"spinlock_wait_cycles":4224,"thread_start_us":63,"threads_started":1}
I20260809 03:30:33.824447 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.819901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47157 (local address 127.15.222.66:42195)
0809 03:30:33.820017 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.820021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.820183 (+ 162us) server_negotiation.cc:407] Connection header received
0809 03:30:33.820329 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.820332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.820380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.820464 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:33.820777 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.821553 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.823161 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.823776 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.823887 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.823946 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.824264 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.824342 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:33.824599 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.819722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42195 (local address 127.15.222.65:47157)
0809 03:30:33.820095 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:33.820110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:33.820227 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:33.820487 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:33.820490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:33.820517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:33.820684 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.820690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.821656 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.821660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.823029 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:33.823041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.824013 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.824017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:33.824123 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.824145 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:33.824223 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:30:33.824292 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":103,"spinlock_wait_cycles":3072,"thread_start_us":90,"threads_started":1}
I20260809 03:30:33.825075 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9a9de3e4c14c3d916b604c4fac077d" candidate_uuid: "001906698b2441979f3f786812886e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557595f6ed2d40b9840fd838e7feadf9" is_pre_election: true
I20260809 03:30:33.825150 28065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9a9de3e4c14c3d916b604c4fac077d" candidate_uuid: "001906698b2441979f3f786812886e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d864ffd4065481ead0362cc6fc23f3d" is_pre_election: true
I20260809 03:30:33.825352 28303 raft_consensus.cc:2454] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001906698b2441979f3f786812886e37 in term 0.
I20260809 03:30:33.825416 28065 raft_consensus.cc:2454] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001906698b2441979f3f786812886e37 in term 0.
I20260809 03:30:33.825802 27753 leader_election.cc:302] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001906698b2441979f3f786812886e37, 557595f6ed2d40b9840fd838e7feadf9; no voters:
I20260809 03:30:33.826071 28430 raft_consensus.cc:2781] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:33.826150 28430 raft_consensus.cc:482] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:33.826181 28430 raft_consensus.cc:3037] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:33.827193 28430 raft_consensus.cc:504] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.827504 28430 leader_election.cc:288] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [CANDIDATE]: Term 1 election: Requested vote from peers 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765), 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195)
I20260809 03:30:33.827796 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9a9de3e4c14c3d916b604c4fac077d" candidate_uuid: "001906698b2441979f3f786812886e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557595f6ed2d40b9840fd838e7feadf9"
I20260809 03:30:33.827826 28065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9a9de3e4c14c3d916b604c4fac077d" candidate_uuid: "001906698b2441979f3f786812886e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d864ffd4065481ead0362cc6fc23f3d"
I20260809 03:30:33.827904 28303 raft_consensus.cc:3037] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:33.827920 28065 raft_consensus.cc:3037] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:33.828892 28065 raft_consensus.cc:2454] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001906698b2441979f3f786812886e37 in term 1.
I20260809 03:30:33.829098 28303 raft_consensus.cc:2454] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001906698b2441979f3f786812886e37 in term 1.
I20260809 03:30:33.829137 27755 leader_election.cc:302] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001906698b2441979f3f786812886e37, 2d864ffd4065481ead0362cc6fc23f3d; no voters:
I20260809 03:30:33.829299 28430 raft_consensus.cc:2781] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:33.829501 28430 raft_consensus.cc:686] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 1 LEADER]: Becoming Leader. State: Replica: 001906698b2441979f3f786812886e37, State: Running, Role: LEADER
I20260809 03:30:33.829850 28430 consensus_queue.cc:237] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } }
I20260809 03:30:33.833559 26387 catalog_manager.cc:5495] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001906698b2441979f3f786812886e37 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "001906698b2441979f3f786812886e37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:33.868247 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.864688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.15.222.65:44941)
0809 03:30:33.864807 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.864812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.864826 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:33.864980 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.864983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.865038 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.865127 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:33.865404 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.865976 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.866819 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.867048 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.867139 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.867446 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:33.867863 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:33.867980 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.868051 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.868109 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:30:33.873910 28303 raft_consensus.cc:1261] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Refusing update from remote peer 001906698b2441979f3f786812886e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:33.873909 28065 raft_consensus.cc:1261] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Refusing update from remote peer 001906698b2441979f3f786812886e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:33.874616 28430 consensus_queue.cc:1035] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:33.874815 28440 consensus_queue.cc:1035] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:30:33.883214 28377 tablet.cc:2369] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:33.884788 28447 mvcc.cc:205] Tried to move back new op lower bound from 7316464573940674560 to 7316464573769797632. Current Snapshot: MvccSnapshot[applied={T|T < 7316464573940674560}]
I20260809 03:30:33.893441 26385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:33.893576 26385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:33.897185 26385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcf35202e96c43009c3150248750bfad: Sending DeleteTablet for 3 replicas of tablet ff9a9de3e4c14c3d916b604c4fac077d
I20260809 03:30:33.898033 28263 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:33 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58988
I20260809 03:30:33.898552 28033 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:33 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:37256
I20260809 03:30:33.898847 27824 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9a9de3e4c14c3d916b604c4fac077d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:33 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:34972
I20260809 03:30:33.899278 28459 tablet_replica.cc:330] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: stopping tablet replica
I20260809 03:30:33.899711 28459 raft_consensus.cc:2229] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:33.899838 28458 tablet_replica.cc:330] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: stopping tablet replica
I20260809 03:30:33.900141 28458 raft_consensus.cc:2229] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:33.900158 28459 raft_consensus.cc:2258] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 03:30:33.900527 27400 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.15.222.125:41829: Invalid argument: socket address is ambiguous
I20260809 03:30:33.900456 28458 raft_consensus.cc:2258] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:33.901158 28457 tablet_replica.cc:330] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: stopping tablet replica
I20260809 03:30:33.901412 28457 raft_consensus.cc:2229] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:33.901677 28457 raft_consensus.cc:2258] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:33.901983 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 26333
I20260809 03:30:33.902122 28459 ts_tablet_manager.cc:1905] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:33.905964 28458 ts_tablet_manager.cc:1905] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:33.907158 28459 ts_tablet_manager.cc:1918] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:33.907320 28459 log.cc:1196] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/ff9a9de3e4c14c3d916b604c4fac077d
I20260809 03:30:33.907727 28459 ts_tablet_manager.cc:1939] T ff9a9de3e4c14c3d916b604c4fac077d P 001906698b2441979f3f786812886e37: Deleting consensus metadata
I20260809 03:30:33.909181 28458 ts_tablet_manager.cc:1918] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:33.909319 28458 log.cc:1196] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/ff9a9de3e4c14c3d916b604c4fac077d
I20260809 03:30:33.909751 28458 ts_tablet_manager.cc:1939] T ff9a9de3e4c14c3d916b604c4fac077d P 2d864ffd4065481ead0362cc6fc23f3d: Deleting consensus metadata
I20260809 03:30:33.919610 28457 ts_tablet_manager.cc:1905] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:33.919672 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.900673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.15.222.125:41829)
0809 03:30:33.900794 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.900799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.911399 (+ 10600us) server_negotiation.cc:407] Connection header received
0809 03:30:33.911544 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.911547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.911598 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.911678 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.912712 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.913544 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.914474 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.914715 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.914814 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.916166 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.916170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.916173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.916198 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.917545 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.918022 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.918026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.918027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.918258 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.918603 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.918607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.918608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.918836 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.918957 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.919353 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.919521 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:30:33.922505 28457 ts_tablet_manager.cc:1918] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:33.922613 28457 log.cc:1196] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/ff9a9de3e4c14c3d916b604c4fac077d
I20260809 03:30:33.922924 28457 ts_tablet_manager.cc:1939] T ff9a9de3e4c14c3d916b604c4fac077d P 557595f6ed2d40b9840fd838e7feadf9: Deleting consensus metadata
W20260809 03:30:33.923380 27964 connection.cc:535] server connection from 127.0.0.1:37256 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 03:30:33.923834 27755 connection.cc:535] server connection from 127.0.0.1:34972 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 03:30:33.924211 28180 connection.cc:441] server connection from 127.0.0.1:58988 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58988 (request call id 1) could send its response
I20260809 03:30:33.930969 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.900763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.15.222.124:39739)
0809 03:30:33.900857 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.900862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.921099 (+ 20237us) server_negotiation.cc:407] Connection header received
0809 03:30:33.921322 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.921325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.921382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.921434 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.922310 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.923077 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.925161 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.925479 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.925572 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.927294 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.927298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.927301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.927328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.928885 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.929333 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.929337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.929339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.929426 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.929752 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.929755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.929757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.929970 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.930081 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.930682 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.930807 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:33.935642 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.926557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.15.222.124:39739)
0809 03:30:33.926663 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.926676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.927150 (+ 474us) server_negotiation.cc:407] Connection header received
0809 03:30:33.927310 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.927313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.927355 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.927413 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.928175 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.928929 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.930104 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.930453 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.930521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.932437 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.932440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.932443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.932466 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.934039 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.934439 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.934442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.934444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.934523 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.934752 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.934756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.934758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.934945 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.935085 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.935374 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.935484 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:33.937369 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:33.928400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.15.222.125:41829)
0809 03:30:33.928507 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:33.928511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:33.928888 (+ 377us) server_negotiation.cc:407] Connection header received
0809 03:30:33.929047 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:33.929050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:33.929093 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:33.929142 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:33.929975 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.930653 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.931881 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:33.932147 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:33.932205 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:33.934114 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:33.934118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:33.934121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:33.934146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:33.935725 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.936176 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.936179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.936181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.936268 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:33.936555 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:33.936558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:33.936560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:33.936770 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:33.936895 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:33.937113 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 03:30:33.937221 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:34.888230 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.887737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:40515)
0809 03:30:34.888006 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.888113 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260809 03:30:34.888577 27929 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:41409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:34.889973 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.889574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:44257)
0809 03:30:34.889811 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.889885 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
W20260809 03:30:34.890462 28145 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:41409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:34.890473 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:34.889879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:36525)
0809 03:30:34.890290 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:34.890371 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":60,"threads_started":1}
W20260809 03:30:34.890923 28374 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:41409 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.398336 28482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcf35202e96c43009c3150248750bfad)
I20260809 03:30:35.398485 28482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } }
I20260809 03:30:35.398990 28482 leader_election.cc:288] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcf35202e96c43009c3150248750bfad (127.15.222.126:41409), 344276192d594e56be9aeea2cf783269 (127.15.222.124:39739)
I20260809 03:30:35.399020 28483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcf35202e96c43009c3150248750bfad)
I20260809 03:30:35.399236 28483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } }
I20260809 03:30:35.399619 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.399128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44872)
0809 03:30:35.399429 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.399518 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":65,"threads_started":1}
W20260809 03:30:35.399749 27324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.400355 28483 leader_election.cc:288] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcf35202e96c43009c3150248750bfad (127.15.222.126:41409), 4c379d4bee144e8babd3ea195e61c613 (127.15.222.125:41829)
I20260809 03:30:35.400563 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.400261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44880)
0809 03:30:35.400454 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.400501 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260809 03:30:35.400689 27324 leader_election.cc:334] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcf35202e96c43009c3150248750bfad (127.15.222.126:41409): Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.400997 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.400432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44884)
0809 03:30:35.400835 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.400915 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":62,"threads_started":1}
W20260809 03:30:35.401199 27610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.401985 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.401773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44898)
0809 03:30:35.401867 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.401927 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:30:35.402081 27610 leader_election.cc:334] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcf35202e96c43009c3150248750bfad (127.15.222.126:41409): Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.403383 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.399470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.15.222.124:39739)
0809 03:30:35.399682 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.399687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.399703 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:30:35.399926 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.399929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.399977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.400107 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.400414 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.401171 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.402508 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.402926 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.403015 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.403060 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.403237 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.403288 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260809 03:30:35.403425 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.399200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:39739 (local address 127.0.0.1:42306)
0809 03:30:35.399531 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.399543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.399637 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.400134 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.400137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.400157 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.400267 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.400274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.401274 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.401277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.402401 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.402411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.403033 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.403036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.403125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.403152 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.403230 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.403302 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260809 03:30:35.403867 27644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c379d4bee144e8babd3ea195e61c613" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "344276192d594e56be9aeea2cf783269" is_pre_election: true
I20260809 03:30:35.404070 27644 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c379d4bee144e8babd3ea195e61c613 in term 1.
I20260809 03:30:35.404366 27329 leader_election.cc:302] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 344276192d594e56be9aeea2cf783269, 4c379d4bee144e8babd3ea195e61c613; no voters: dcf35202e96c43009c3150248750bfad
I20260809 03:30:35.404513 28482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 03:30:35.404569 28482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dcf35202e96c43009c3150248750bfad)
I20260809 03:30:35.404615 28482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:30:35.405073 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.400912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.15.222.125:41829)
0809 03:30:35.401150 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.401158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.401609 (+ 451us) server_negotiation.cc:407] Connection header received
0809 03:30:35.401733 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.401737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.401777 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.401852 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.402171 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.402768 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.404064 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.404600 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.404710 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.404750 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.404934 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.404990 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260809 03:30:35.405112 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.400708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:41829 (local address 127.0.0.1:41448)
0809 03:30:35.401519 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.401532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.401630 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.401953 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.401956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.401973 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.402056 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.402061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.402873 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.402875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.403959 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.403970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.404719 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.404722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.404807 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.404830 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.404923 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.404986 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260809 03:30:35.405506 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "344276192d594e56be9aeea2cf783269" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4c379d4bee144e8babd3ea195e61c613" is_pre_election: true
I20260809 03:30:35.405896 28482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } }
I20260809 03:30:35.406114 27364 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 344276192d594e56be9aeea2cf783269 in current term 2: Already voted for candidate 4c379d4bee144e8babd3ea195e61c613 in this term.
I20260809 03:30:35.406200 28482 leader_election.cc:288] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 election: Requested vote from peers dcf35202e96c43009c3150248750bfad (127.15.222.126:41409), 344276192d594e56be9aeea2cf783269 (127.15.222.124:39739)
I20260809 03:30:35.406491 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.406296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44908)
0809 03:30:35.406383 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.406434 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 03:30:35.406555 27610 leader_election.cc:302] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344276192d594e56be9aeea2cf783269; no voters: 4c379d4bee144e8babd3ea195e61c613, dcf35202e96c43009c3150248750bfad
I20260809 03:30:35.406576 27644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c379d4bee144e8babd3ea195e61c613" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "344276192d594e56be9aeea2cf783269"
I20260809 03:30:35.406677 27644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:30:35.407541 27644 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c379d4bee144e8babd3ea195e61c613 in term 2.
I20260809 03:30:35.407640 28483 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:30:35.407845 27329 leader_election.cc:302] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344276192d594e56be9aeea2cf783269, 4c379d4bee144e8babd3ea195e61c613; no voters:
I20260809 03:30:35.407998 28482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:30:35.408190 28482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [term 2 LEADER]: Becoming Leader. State: Replica: 4c379d4bee144e8babd3ea195e61c613, State: Running, Role: LEADER
I20260809 03:30:35.408290 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.408063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44914)
0809 03:30:35.408189 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.408236 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260809 03:30:35.408366 27324 leader_election.cc:334] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dcf35202e96c43009c3150248750bfad (127.15.222.126:41409): Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
I20260809 03:30:35.408464 28482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } }
I20260809 03:30:35.409648 28500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c379d4bee144e8babd3ea195e61c613. Latest consensus state: current_term: 2 leader_uuid: "4c379d4bee144e8babd3ea195e61c613" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:35.409765 28500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:35.410022 28504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:35.412040 28504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:35.412410 28504 catalog_manager.cc:1259] Loaded cluster ID: f4f4b7b1086a4ad0a66cc5856b5a7c7a
I20260809 03:30:35.412454 28504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:35.412953 28504 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:35.413316 28504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Loaded TSK: 0
I20260809 03:30:35.413975 28504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:35.443635 27338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:35.446627 27338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:35.449338 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.449122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44930)
0809 03:30:35.449223 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.449276 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 03:30:35.449397 27644 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [term 2 FOLLOWER]: Refusing update from remote peer 4c379d4bee144e8babd3ea195e61c613: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 03:30:35.449848 28482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 03:30:35.450042 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.449823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.0.0.1:44932)
0809 03:30:35.449930 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.449979 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:30:35.450136 27324 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 -> Peer dcf35202e96c43009c3150248750bfad (127.15.222.126:41409): Couldn't send request to peer dcf35202e96c43009c3150248750bfad. Status: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 03:30:35.451184 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c379d4bee144e8babd3ea195e61c613. Latest consensus state: current_term: 2 leader_uuid: "4c379d4bee144e8babd3ea195e61c613" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:35.451293 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:35.452373 28482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4c379d4bee144e8babd3ea195e61c613" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:35.452489 28482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:35.452697 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4c379d4bee144e8babd3ea195e61c613" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "dcf35202e96c43009c3150248750bfad" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41409 } } peers { permanent_uuid: "4c379d4bee144e8babd3ea195e61c613" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 41829 } attrs { promote: false } } peers { permanent_uuid: "344276192d594e56be9aeea2cf783269" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 39739 } attrs { promote: false } } }
I20260809 03:30:35.452803 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344276192d594e56be9aeea2cf783269 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:30:35.453018 28506 mvcc.cc:205] Tried to move back new op lower bound from 7316464580396613632 to 7316464580235231232. Current Snapshot: MvccSnapshot[applied={T|T < 7316464580396613632}]
I20260809 03:30:35.461755 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.457407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42195 (local address 127.0.0.1:37270)
0809 03:30:35.457502 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.457516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.457592 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.458304 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.458308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.458326 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.458448 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.458455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.459521 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.459524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.460301 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.460311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.461259 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.461263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.461445 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.461469 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.461537 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.461604 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":78}
I20260809 03:30:35.462113 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.457543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.15.222.66:42195)
0809 03:30:35.457832 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.457837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.457850 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:35.457900 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.457903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.458178 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.458577 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.458701 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.459401 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.460417 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.461043 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.461372 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.461442 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.461908 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.461991 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
I20260809 03:30:35.462307 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.457639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:37765 (local address 127.0.0.1:59002)
0809 03:30:35.457801 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.457815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.457908 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.458394 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.458397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.458417 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.458545 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.458553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.459583 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.459586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.460778 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.460788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.461967 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.461970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.462064 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.462088 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.462133 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.462195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":23}
I20260809 03:30:35.462620 28033 tablet_service.cc:1467] Processing CreateTablet for tablet bfeed461805b4944a5c62eb6c6500c94 (DEFAULT_TABLE table=test-table [id=434cb3087a47424989d092a49b3284bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:35.462755 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.457903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.15.222.67:37765)
0809 03:30:35.458154 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.458159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.458173 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:35.458224 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.458227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.458271 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.458649 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.458758 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.459462 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.460887 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.461840 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.462482 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.462541 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.462573 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.462634 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260809 03:30:35.462936 28033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeed461805b4944a5c62eb6c6500c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:35.463243 28263 tablet_service.cc:1467] Processing CreateTablet for tablet bfeed461805b4944a5c62eb6c6500c94 (DEFAULT_TABLE table=test-table [id=434cb3087a47424989d092a49b3284bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:35.463493 28263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeed461805b4944a5c62eb6c6500c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:35.465080 28515 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Bootstrap starting.
I20260809 03:30:35.465746 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.457965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44941 (local address 127.0.0.1:34992)
0809 03:30:35.458969 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.458983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.459071 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.459291 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.459293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.459306 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.459391 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.459396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.460356 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.460359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.461405 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.461415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.465355 (+ 3940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.465359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.465442 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.465464 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.465530 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.465597 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260809 03:30:35.465901 28517 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Bootstrap starting.
I20260809 03:30:35.466183 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.458309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.15.222.65:44941)
0809 03:30:35.458570 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.458574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.459061 (+ 487us) server_negotiation.cc:407] Connection header received
0809 03:30:35.459149 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.459152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.459201 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.459279 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.459490 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.460247 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.461542 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.462156 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.465227 (+ 3071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.465293 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.465969 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.466060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260809 03:30:35.466552 28515 tablet_bootstrap.cc:654] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:35.466637 28517 tablet_bootstrap.cc:654] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:35.466718 27824 tablet_service.cc:1467] Processing CreateTablet for tablet bfeed461805b4944a5c62eb6c6500c94 (DEFAULT_TABLE table=test-table [id=434cb3087a47424989d092a49b3284bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:35.466985 27824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfeed461805b4944a5c62eb6c6500c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:35.467653 28517 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: No bootstrap required, opened a new log
I20260809 03:30:35.467722 28517 ts_tablet_manager.cc:1397] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:35.468073 28517 raft_consensus.cc:348] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.468186 28517 raft_consensus.cc:374] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:35.468212 28517 raft_consensus.cc:729] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557595f6ed2d40b9840fd838e7feadf9, State: Initialized, Role: FOLLOWER
I20260809 03:30:35.468328 28517 consensus_queue.cc:260] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.468600 28517 ts_tablet_manager.cc:1428] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:35.469413 28522 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37: Bootstrap starting.
I20260809 03:30:35.469518 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.468928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:46977)
0809 03:30:35.469306 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.469361 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":82,"threads_started":1}
I20260809 03:30:35.470286 28522 tablet_bootstrap.cc:654] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:35.470316 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.469969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:40865)
0809 03:30:35.470203 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.470249 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260809 03:30:35.471130 28376 heartbeater.cc:503] Master 127.15.222.125:41829 was elected leader, sending a full tablet report...
I20260809 03:30:35.471619 28515 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: No bootstrap required, opened a new log
I20260809 03:30:35.471701 28515 ts_tablet_manager.cc:1397] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260809 03:30:35.471752 28522 tablet_bootstrap.cc:492] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37: No bootstrap required, opened a new log
I20260809 03:30:35.471843 28522 ts_tablet_manager.cc:1397] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:35.472108 28515 raft_consensus.cc:348] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.472200 28515 raft_consensus.cc:374] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:35.472229 28515 raft_consensus.cc:729] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d864ffd4065481ead0362cc6fc23f3d, State: Initialized, Role: FOLLOWER
I20260809 03:30:35.472281 28522 raft_consensus.cc:348] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.472355 28515 consensus_queue.cc:260] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.472405 28522 raft_consensus.cc:374] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:35.472445 28522 raft_consensus.cc:729] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001906698b2441979f3f786812886e37, State: Initialized, Role: FOLLOWER
I20260809 03:30:35.472563 28522 consensus_queue.cc:260] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.472673 28515 ts_tablet_manager.cc:1428] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:35.472880 28522 ts_tablet_manager.cc:1428] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:35.474186 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.473236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:33163)
0809 03:30:35.474045 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.474100 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":95,"threads_started":1}
I20260809 03:30:35.475046 27930 heartbeater.cc:503] Master 127.15.222.125:41829 was elected leader, sending a full tablet report...
I20260809 03:30:35.476116 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.473297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:57349)
0809 03:30:35.473520 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.473577 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260809 03:30:35.476680 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.476491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:56295)
0809 03:30:35.476586 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.476631 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 03:30:35.625361 28521 raft_consensus.cc:482] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:35.625533 28521 raft_consensus.cc:504] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.626693 28521 leader_election.cc:288] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195), 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
I20260809 03:30:35.630354 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.626957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44941 (local address 127.15.222.67:40769)
0809 03:30:35.627286 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.627299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.627395 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.627633 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.627636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.627650 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.627730 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.627736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.628408 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.628411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.629302 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.629311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.629973 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.630066 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.630083 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.630152 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.630224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":255}
I20260809 03:30:35.630359 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.627045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:40769 (local address 127.15.222.65:44941)
0809 03:30:35.627160 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.627164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.627384 (+ 220us) server_negotiation.cc:407] Connection header received
0809 03:30:35.627486 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.627489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.627539 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.627615 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.627828 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.628306 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629424 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.629851 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629967 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.630017 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.630176 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.630245 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:35.630353 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.627005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:47733 (local address 127.15.222.66:42195)
0809 03:30:35.627124 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.627128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.627190 (+ 62us) server_negotiation.cc:407] Connection header received
0809 03:30:35.627349 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.627353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.627419 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.627489 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:35.627767 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.628281 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629439 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.629837 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.629990 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.630155 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.630221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:35.630409 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.626943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42195 (local address 127.15.222.67:47733)
0809 03:30:35.627089 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.627104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:35.627238 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:35.627508 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:35.627511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:35.627527 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:35.627659 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.627666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.628402 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.628405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.629303 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:35.629313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.629957 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.629960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:35.630044 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.630066 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:35.630127 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:30:35.630201 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":65}
I20260809 03:30:35.630976 27857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeed461805b4944a5c62eb6c6500c94" candidate_uuid: "557595f6ed2d40b9840fd838e7feadf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001906698b2441979f3f786812886e37" is_pre_election: true
I20260809 03:30:35.631103 28065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeed461805b4944a5c62eb6c6500c94" candidate_uuid: "557595f6ed2d40b9840fd838e7feadf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d864ffd4065481ead0362cc6fc23f3d" is_pre_election: true
I20260809 03:30:35.631202 27857 raft_consensus.cc:2454] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557595f6ed2d40b9840fd838e7feadf9 in term 0.
I20260809 03:30:35.631224 28065 raft_consensus.cc:2454] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557595f6ed2d40b9840fd838e7feadf9 in term 0.
I20260809 03:30:35.631628 28179 leader_election.cc:302] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001906698b2441979f3f786812886e37, 557595f6ed2d40b9840fd838e7feadf9; no voters:
I20260809 03:30:35.631871 28521 raft_consensus.cc:2781] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:35.631938 28521 raft_consensus.cc:482] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:35.631974 28521 raft_consensus.cc:3037] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:35.632968 28521 raft_consensus.cc:504] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.633284 28521 leader_election.cc:288] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195), 001906698b2441979f3f786812886e37 (127.15.222.65:44941)
I20260809 03:30:35.633579 27857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeed461805b4944a5c62eb6c6500c94" candidate_uuid: "557595f6ed2d40b9840fd838e7feadf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001906698b2441979f3f786812886e37"
I20260809 03:30:35.633607 28065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfeed461805b4944a5c62eb6c6500c94" candidate_uuid: "557595f6ed2d40b9840fd838e7feadf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d864ffd4065481ead0362cc6fc23f3d"
I20260809 03:30:35.633684 27857 raft_consensus.cc:3037] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:35.633731 28065 raft_consensus.cc:3037] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:35.634626 27857 raft_consensus.cc:2454] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557595f6ed2d40b9840fd838e7feadf9 in term 1.
I20260809 03:30:35.634789 28065 raft_consensus.cc:2454] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557595f6ed2d40b9840fd838e7feadf9 in term 1.
I20260809 03:30:35.634930 28179 leader_election.cc:302] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001906698b2441979f3f786812886e37, 557595f6ed2d40b9840fd838e7feadf9; no voters:
I20260809 03:30:35.635090 28521 raft_consensus.cc:2781] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:35.635294 28521 raft_consensus.cc:686] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 1 LEADER]: Becoming Leader. State: Replica: 557595f6ed2d40b9840fd838e7feadf9, State: Running, Role: LEADER
I20260809 03:30:35.635628 28521 consensus_queue.cc:237] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } }
I20260809 03:30:35.635885 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.635687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:58347)
0809 03:30:35.635783 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.635832 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 03:30:35.638049 27348 catalog_manager.cc:5495] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67). New cstate: current_term: 1 leader_uuid: "557595f6ed2d40b9840fd838e7feadf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557595f6ed2d40b9840fd838e7feadf9" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:35.649744 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.646581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.15.222.67:37765)
0809 03:30:35.646685 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:35.646690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:35.646703 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:35.646853 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:35.646855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:35.646905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:35.646987 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:35.647211 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.647674 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.648465 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:35.648666 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:35.648747 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:35.648998 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:35.649401 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:35.649496 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:35.649567 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:30:35.649615 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:30:35.652029 28065 raft_consensus.cc:1261] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Refusing update from remote peer 557595f6ed2d40b9840fd838e7feadf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:35.652236 27857 raft_consensus.cc:1261] T bfeed461805b4944a5c62eb6c6500c94 P 001906698b2441979f3f786812886e37 [term 1 FOLLOWER]: Refusing update from remote peer 557595f6ed2d40b9840fd838e7feadf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:35.652452 28532 consensus_queue.cc:1035] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d864ffd4065481ead0362cc6fc23f3d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:35.652686 28521 consensus_queue.cc:1035] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001906698b2441979f3f786812886e37" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:35.653534 28146 heartbeater.cc:503] Master 127.15.222.125:41829 was elected leader, sending a full tablet report...
I20260809 03:30:35.654176 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.653966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:34969)
0809 03:30:35.654060 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.654113 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 03:30:35.654439 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.654171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:52971)
0809 03:30:35.654318 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.654371 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 03:30:35.657202 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.656911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:42119)
0809 03:30:35.657092 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.657140 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260809 03:30:35.658370 28536 mvcc.cc:205] Tried to move back new op lower bound from 7316464581226799104 to 7316464581165916160. Current Snapshot: MvccSnapshot[applied={T|T < 7316464581226799104 or (T in {7316464581226799104})}]
I20260809 03:30:35.659163 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.658965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.66:41271)
0809 03:30:35.659048 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.659098 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 03:30:35.661046 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.660709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.65:35139)
0809 03:30:35.660927 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.660981 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260809 03:30:35.661280 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.660989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41409 (local address 127.15.222.67:44321)
0809 03:30:35.661180 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.661225 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260809 03:30:35.661620 28539 mvcc.cc:205] Tried to move back new op lower bound from 7316464581226799104 to 7316464581165916160. Current Snapshot: MvccSnapshot[applied={T|T < 7316464581226799104 or (T in {7316464581226799104})}]
I20260809 03:30:35.665535 27348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:35.665683 27348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:35.669301 27348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c379d4bee144e8babd3ea195e61c613: Sending DeleteTablet for 3 replicas of tablet bfeed461805b4944a5c62eb6c6500c94
I20260809 03:30:35.670064 28263 tablet_service.cc:1514] Processing DeleteTablet for tablet bfeed461805b4944a5c62eb6c6500c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:35 UTC) from {username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:59002
I20260809 03:30:35.670373 28550 tablet_replica.cc:330] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: stopping tablet replica
I20260809 03:30:35.670560 28550 raft_consensus.cc:2229] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:35.670732 28033 tablet_service.cc:1514] Processing DeleteTablet for tablet bfeed461805b4944a5c62eb6c6500c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:35 UTC) from {username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:37270
I20260809 03:30:35.670854 27824 tablet_service.cc:1514] Processing DeleteTablet for tablet bfeed461805b4944a5c62eb6c6500c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:35 UTC) from {username='oryx', principal='oryx/127.15.222.125@KRBTEST.COM'} at 127.0.0.1:34992
I20260809 03:30:35.671001 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 27574
I20260809 03:30:35.681937 28550 raft_consensus.cc:2258] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:35.682816 28550 ts_tablet_manager.cc:1905] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:35.685938 28550 ts_tablet_manager.cc:1918] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:35.686100 28550 log.cc:1196] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/bfeed461805b4944a5c62eb6c6500c94
I20260809 03:30:35.686451 28550 ts_tablet_manager.cc:1939] T bfeed461805b4944a5c62eb6c6500c94 P 557595f6ed2d40b9840fd838e7feadf9: Deleting consensus metadata
I20260809 03:30:35.688211 27329 catalog_manager.cc:4841] TS 557595f6ed2d40b9840fd838e7feadf9 (127.15.222.67:37765): tablet bfeed461805b4944a5c62eb6c6500c94 (table test-table [id=434cb3087a47424989d092a49b3284bd]) successfully deleted
I20260809 03:30:35.691359 28552 tablet_replica.cc:330] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: stopping tablet replica
I20260809 03:30:35.691517 28552 raft_consensus.cc:2229] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:35.691663 28552 raft_consensus.cc:2258] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:35.692422 28552 ts_tablet_manager.cc:1905] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:35.694974 28552 ts_tablet_manager.cc:1918] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:35.695053 28552 log.cc:1196] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/bfeed461805b4944a5c62eb6c6500c94
I20260809 03:30:35.695354 28552 ts_tablet_manager.cc:1939] T bfeed461805b4944a5c62eb6c6500c94 P 2d864ffd4065481ead0362cc6fc23f3d: Deleting consensus metadata
I20260809 03:30:35.696091 27330 catalog_manager.cc:4841] TS 2d864ffd4065481ead0362cc6fc23f3d (127.15.222.66:42195): tablet bfeed461805b4944a5c62eb6c6500c94 (table test-table [id=434cb3087a47424989d092a49b3284bd]) successfully deleted
I20260809 03:30:35.698616 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 27937
I20260809 03:30:35.698863 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:35.698518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44941 (local address 127.0.0.1:35008)
0809 03:30:35.698707 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:35.698776 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260809 03:30:35.699066 27324 catalog_manager.cc:4584] TS 001906698b2441979f3f786812886e37 (127.15.222.65:44941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfeed461805b4944a5c62eb6c6500c94: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44941: connect: Connection refused (error 111)
I20260809 03:30:35.709977 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 28155
I20260809 03:30:35.722512 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 27137
I20260809 03:30:35.734732 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 27302
2026-08-09T03:30:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5479 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:35 dist-test-slave-cvwc krb5kdc[28558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:35 dist-test-slave-cvwc krb5kdc[28558](info): set up 2 sockets
Aug 09 03:30:35 dist-test-slave-cvwc krb5kdc[28558](info): commencing operation
krb5kdc: starting...
W20260809 03:30:36.875494 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:36 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab.
Entry for principal oryx/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab.
I20260809 03:30:36.988286 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38087
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab
--principal=oryx/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:37.097034 28576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:37.097299 28576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:37.097355 28576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:37.101308 28576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:37.101366 28576 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 03:30:37.101392 28576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:37.101413 28576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:37.101436 28576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:37.101465 28576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:37.106623 28576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.126.keytab
--principal=oryx/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:37.107602 28576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:37.108583 28576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:37.116780 28582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.116806 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.116806 28583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:37.117017 28576 server_base.cc:1034] running on GCE node
I20260809 03:30:37.117651 28576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:37.118703 28576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:37.119886 28576 hybrid_clock.cc:648] HybridClock initialized: now 1786246237119868 us; error 27 us; skew 500 ppm
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:37.122709 28576 init.cc:377] Logged in from keytab as oryx/127.15.222.126@KRBTEST.COM (short username oryx)
I20260809 03:30:37.123610 28576 webserver.cc:460] Webserver started at http://127.15.222.126:45895/ using document root <none> and password file <none>
I20260809 03:30:37.124150 28576 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:37.124218 28576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:37.124440 28576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:37.126509 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "c99419ca5d354519b0cc1a622fae2b3b"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "72501a17dd6086eb3bb482c3b1bbdf1a"
server_key_iv: "25fac227afbe26a46a3fcd13e7b2fb04"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.127058 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "c99419ca5d354519b0cc1a622fae2b3b"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "72501a17dd6086eb3bb482c3b1bbdf1a"
server_key_iv: "25fac227afbe26a46a3fcd13e7b2fb04"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.130700 28576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:37.133044 28600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.134267 28576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:37.134380 28576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "c99419ca5d354519b0cc1a622fae2b3b"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "72501a17dd6086eb3bb482c3b1bbdf1a"
server_key_iv: "25fac227afbe26a46a3fcd13e7b2fb04"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.134487 28576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:37.156780 28576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:37.159708 28576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:37.159919 28576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:37.168440 28576 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38087
I20260809 03:30:37.168495 28699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38087 every 8 connection(s)
I20260809 03:30:37.169440 28576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:37.172224 28701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:37.173795 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 28576
I20260809 03:30:37.173903 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:37.174192 16249 external_mini_cluster.cc:1428] Setting key 587a303df74aacc1119ea8e99b91f530
I20260809 03:30:37.178282 28701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Bootstrap starting.
I20260809 03:30:37.180451 28701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:37.181231 28701 log.cc:824] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:37.183077 28701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: No bootstrap required, opened a new log
I20260809 03:30:37.186159 28701 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } }
I20260809 03:30:37.186395 28701 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:37.186468 28701 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99419ca5d354519b0cc1a622fae2b3b, State: Initialized, Role: FOLLOWER
I20260809 03:30:37.186923 28701 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } }
I20260809 03:30:37.187086 28701 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:37.187155 28701 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:37.187251 28701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:37.187827 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.175652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.15.222.126:38087)
0809 03:30:37.176078 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.176092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.176128 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:30:37.176723 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.176738 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.177058 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.177362 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.178158 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.179220 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.180031 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.180479 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.180617 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.182993 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.182997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.183011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.183038 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.185127 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.185530 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.185535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.185540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.185650 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.185925 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.185928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.185930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.186304 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.186500 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.186780 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.187054 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260809 03:30:37.188107 28701 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } }
I20260809 03:30:37.188426 28701 leader_election.cc:302] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99419ca5d354519b0cc1a622fae2b3b; no voters:
I20260809 03:30:37.188706 28701 leader_election.cc:288] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:37.188802 28708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:37.189031 28708 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [term 1 LEADER]: Becoming Leader. State: Replica: c99419ca5d354519b0cc1a622fae2b3b, State: Running, Role: LEADER
I20260809 03:30:37.189350 28708 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } }
I20260809 03:30:37.189973 28701 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:37.191087 28710 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99419ca5d354519b0cc1a622fae2b3b. Latest consensus state: current_term: 1 leader_uuid: "c99419ca5d354519b0cc1a622fae2b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } } }
I20260809 03:30:37.191098 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99419ca5d354519b0cc1a622fae2b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99419ca5d354519b0cc1a622fae2b3b" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38087 } } }
I20260809 03:30:37.191231 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:37.191229 28710 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:37.191668 28721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:37.194291 28721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:37.200042 28721 catalog_manager.cc:1347] Generated new cluster ID: a4fb9fcbf1d3480cbf89961775eeadf7
I20260809 03:30:37.200107 28721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:37.219447 28721 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:37.220709 28721 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:37.231134 28721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Generated new TSK 0
I20260809 03:30:37.231915 28721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
I20260809 03:30:37.260375 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38087
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:37.371459 28739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:37.371723 28739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:37.371767 28739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:37.375734 28739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:37.375787 28739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:37.375836 28739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:37.375888 28739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:37.381073 28739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:37.382082 28739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:37.383095 28739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:37.392416 28754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.392397 28748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.392421 28749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:37.392745 28739 server_base.cc:1034] running on GCE node
I20260809 03:30:37.393136 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:37.393743 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:37.394907 28739 hybrid_clock.cc:648] HybridClock initialized: now 1786246237394894 us; error 33 us; skew 500 ppm
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:37.397486 28739 init.cc:377] Logged in from keytab as oryx/127.15.222.65@KRBTEST.COM (short username oryx)
I20260809 03:30:37.398329 28739 webserver.cc:460] Webserver started at http://127.15.222.65:41727/ using document root <none> and password file <none>
I20260809 03:30:37.398852 28739 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:37.398927 28739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:37.399134 28739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:37.401260 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "eb3d706b01614f74af26febc495994ca"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "04ed32395c5f92e209d39e39370e4c2e"
server_key_iv: "9984de61d8b64349f2d722659272ab41"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.401855 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "eb3d706b01614f74af26febc495994ca"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "04ed32395c5f92e209d39e39370e4c2e"
server_key_iv: "9984de61d8b64349f2d722659272ab41"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.405368 28739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:37.407704 28764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.408908 28739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:37.409029 28739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "eb3d706b01614f74af26febc495994ca"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "04ed32395c5f92e209d39e39370e4c2e"
server_key_iv: "9984de61d8b64349f2d722659272ab41"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.409129 28739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:37.434542 28739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:37.438119 28739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:37.438316 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:37.439859 28739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:37.439914 28739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.439970 28739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:37.439996 28739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:37.450214 28739 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:33761
I20260809 03:30:37.450222 28956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:33761 every 8 connection(s)
I20260809 03:30:37.451251 28739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:37.452531 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.440638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:49907 (local address 127.15.222.126:38087)
0809 03:30:37.440798 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.440803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.441662 (+ 859us) server_negotiation.cc:407] Connection header received
0809 03:30:37.442672 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.442676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.442735 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.442784 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.444110 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.444870 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.446032 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.446274 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.446372 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.448877 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.448881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.448885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.448917 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.450491 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.450965 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.450969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.450970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.451063 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.451356 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.451360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.451361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.451561 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.451649 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.452216 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.452358 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:30:37.453258 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.441027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.65:49907)
0809 03:30:37.441503 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.441554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.442422 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.443111 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.443117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.443456 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.443968 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.443985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.444998 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.445001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.445921 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.445932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.446477 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.446482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.446678 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.447242 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.447263 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.448722 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.450595 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.450605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.450616 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.450860 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.451142 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.451144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.451145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.451264 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.451661 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.451668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.451974 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.452154 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.452473 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260809 03:30:37.454457 28957 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38087
I20260809 03:30:37.454602 28957 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:37.455147 28957 heartbeater.cc:511] Master 127.15.222.126:38087 requested a full tablet report, sending...
I20260809 03:30:37.456879 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 28739
I20260809 03:30:37.457008 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:37.456962 28628 ts_manager.cc:194] Registered new tserver with Master: eb3d706b01614f74af26febc495994ca (127.15.222.65:33761)
I20260809 03:30:37.457366 16249 external_mini_cluster.cc:1428] Setting key 2ec718137675b8c823f9b4131d246604
I20260809 03:30:37.458511 28628 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:49907
I20260809 03:30:37.464087 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.456739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:45679 (local address 127.15.222.126:38087)
0809 03:30:37.456861 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.456866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.457253 (+ 387us) server_negotiation.cc:407] Connection header received
0809 03:30:37.457349 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.457352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.457393 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.457466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.458065 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.458772 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.459779 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.460003 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.460088 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.461220 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.461222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.461225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.461246 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.462586 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.463014 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.463016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.463018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.463089 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.463361 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.463363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.463364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.463532 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.463623 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.463874 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.463975 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:37.464170 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.456823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.65:45679)
0809 03:30:37.457163 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.457178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.457277 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.457576 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.457579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.457810 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.457938 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.457946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.458946 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.458949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.459671 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.459681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.460100 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.460103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.460187 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.460652 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.460667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.461096 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.462688 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.462691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.462693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.462931 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.463172 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.463174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.463177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.463284 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.463627 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.463630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.463729 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.463834 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.463983 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
I20260809 03:30:37.484110 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38087
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:37.595381 28970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:37.595642 28970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:37.595714 28970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:37.599677 28970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:37.599742 28970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:37.599804 28970 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:37.599862 28970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:37.604909 28970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:37.605958 28970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:37.606953 28970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:37.616178 28970 server_base.cc:1034] running on GCE node
W20260809 03:30:37.616140 28976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.616138 28975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.616129 28978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:37.616729 28970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:37.617290 28970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:37.618438 28970 hybrid_clock.cc:648] HybridClock initialized: now 1786246237618421 us; error 26 us; skew 500 ppm
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:37.620970 28970 init.cc:377] Logged in from keytab as oryx/127.15.222.66@KRBTEST.COM (short username oryx)
I20260809 03:30:37.621804 28970 webserver.cc:460] Webserver started at http://127.15.222.66:43583/ using document root <none> and password file <none>
I20260809 03:30:37.622310 28970 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:37.622366 28970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:37.622536 28970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:37.624459 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "91e7b9fd9bd64c459a270c1f30362a89"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "e2ab238380058b29331b8bca666a2bd4"
server_key_iv: "3468289e98f5ac1bdba1806097321de2"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.624966 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "91e7b9fd9bd64c459a270c1f30362a89"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "e2ab238380058b29331b8bca666a2bd4"
server_key_iv: "3468289e98f5ac1bdba1806097321de2"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.628422 28970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:30:37.630703 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.631815 28970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:37.631929 28970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "91e7b9fd9bd64c459a270c1f30362a89"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "e2ab238380058b29331b8bca666a2bd4"
server_key_iv: "3468289e98f5ac1bdba1806097321de2"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.632028 28970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:37.642306 28970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:37.645133 28970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:37.645290 28970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:37.646678 28970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:37.646726 28970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.646763 28970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:37.646798 28970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:37.656853 28970 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:37705
I20260809 03:30:37.656901 29183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:37705 every 8 connection(s)
I20260809 03:30:37.657670 28970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:37.659544 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.647259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46827 (local address 127.15.222.126:38087)
0809 03:30:37.647365 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.647368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.648333 (+ 965us) server_negotiation.cc:407] Connection header received
0809 03:30:37.649186 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.649190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.649249 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.649329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.650697 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.651494 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.652597 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.652813 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.652911 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.656351 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.656354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.656356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.656382 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.657696 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.658247 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.658250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.658252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.658340 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.658700 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.658702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.658703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.658871 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.658955 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.659339 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.659428 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:30:37.660233 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 28970
I20260809 03:30:37.660372 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:37.660413 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.647756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.66:46827)
0809 03:30:37.648202 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.648239 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.649000 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.649512 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.649520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.650000 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.650541 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.650555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.651623 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.651627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.652482 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.652491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.652982 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.652986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.653187 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.653704 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.653747 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.656200 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.657846 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.657850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.657861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.658138 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.658445 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.658448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.658450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.658622 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.658958 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.658963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.659166 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.659308 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.659543 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260809 03:30:37.660686 16249 external_mini_cluster.cc:1428] Setting key c88109a9aa2fa1031931a1e04c4001fe
I20260809 03:30:37.661646 29185 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38087
I20260809 03:30:37.661811 29185 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:37.662339 29185 heartbeater.cc:511] Master 127.15.222.126:38087 requested a full tablet report, sending...
I20260809 03:30:37.664921 28630 ts_manager.cc:194] Registered new tserver with Master: 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705)
I20260809 03:30:37.665606 28630 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:46827
I20260809 03:30:37.670207 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.663629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:38181 (local address 127.15.222.126:38087)
0809 03:30:37.663729 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.663732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.663941 (+ 209us) server_negotiation.cc:407] Connection header received
0809 03:30:37.664033 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.664036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.664074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.664135 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.664602 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.665131 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.666001 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.666288 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.666381 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.667504 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.667507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.667510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.667534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.668727 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.669097 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.669100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.669102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.669172 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.669415 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.669418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.669420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.669609 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.669691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.669970 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.670065 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:30:37.670223 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.663571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.66:38181)
0809 03:30:37.663857 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.663868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.663960 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.664160 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.664162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.664357 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.664465 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.664472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.665245 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.665248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.665887 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.665906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.666423 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.666426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.666512 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.666950 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.666964 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.667340 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.668823 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.668826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.668828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.669014 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.669251 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.669254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.669256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.669332 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.669698 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.669701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.669818 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.669912 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.670039 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
I20260809 03:30:37.686440 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38087
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:37.795995 29198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:37.796217 29198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:37.796264 29198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:37.800165 29198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:37.800215 29198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:37.800261 29198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 03:30:37.800311 29198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:37.805222 29198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.222.126:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:37.806181 29198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:37.807153 29198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:37.816141 29210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:37.816134 29213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:37.816226 29198 server_base.cc:1034] running on GCE node
W20260809 03:30:37.816138 29211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:37.816738 29198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:37.817301 29198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:37.818454 29198 hybrid_clock.cc:648] HybridClock initialized: now 1786246237818434 us; error 28 us; skew 500 ppm
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:37.820986 29198 init.cc:377] Logged in from keytab as oryx/127.15.222.67@KRBTEST.COM (short username oryx)
I20260809 03:30:37.821833 29198 webserver.cc:460] Webserver started at http://127.15.222.67:36009/ using document root <none> and password file <none>
I20260809 03:30:37.822352 29198 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:37.822427 29198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:37.822623 29198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:37.824517 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "b23f891195a24a1aa460536b0a56da2f"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "39967097c12163b844a1d0b285df2150"
server_key_iv: "f830362f0757ed3247c98c4cc95d1215"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.825026 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "b23f891195a24a1aa460536b0a56da2f"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "39967097c12163b844a1d0b285df2150"
server_key_iv: "f830362f0757ed3247c98c4cc95d1215"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.828428 29198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:30:37.830637 29222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.831740 29198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:37.831841 29198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "b23f891195a24a1aa460536b0a56da2f"
format_stamp: "Formatted at 2026-08-09 03:30:37 on dist-test-slave-cvwc"
server_key: "39967097c12163b844a1d0b285df2150"
server_key_iv: "f830362f0757ed3247c98c4cc95d1215"
server_key_version: "encryptionkey@0"
I20260809 03:30:37.831926 29198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:37.843348 29198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:37.846186 29198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:37.846343 29198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:37.847662 29198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:37.847711 29198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:37.847760 29198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:37.847791 29198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:37.857895 29198 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:46797
I20260809 03:30:37.858809 29198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:37.859637 29412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:46797 every 8 connection(s)
I20260809 03:30:37.860385 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.848349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:38049 (local address 127.15.222.126:38087)
0809 03:30:37.848499 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.848504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.849276 (+ 772us) server_negotiation.cc:407] Connection header received
0809 03:30:37.850209 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.850212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.850269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.850415 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.851896 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.852688 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.853852 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.854156 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.854257 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.856978 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.856981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.856983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.857006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.858347 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.858900 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.858903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.858905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.859005 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.859381 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.859385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.859387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.859615 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.859717 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.860176 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.860271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:37.861105 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.848684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.67:38049)
0809 03:30:37.849118 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.849164 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.850013 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.850662 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.850670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.851122 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.851739 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.851756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.852824 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.852828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.853707 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.853751 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.854421 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.854424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.854556 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.855072 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.855087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.856804 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.858467 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.858484 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.858497 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.858803 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.859088 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.859091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.859093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.859269 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.859740 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.859747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.859979 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.860127 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.860344 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260809 03:30:37.862190 29413 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38087
I20260809 03:30:37.862321 29413 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:37.862329 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 29198
I20260809 03:30:37.862447 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:37.862697 16249 external_mini_cluster.cc:1428] Setting key 13bc5abdeb0b49926e8bfa98aff50b7a
I20260809 03:30:37.862954 29413 heartbeater.cc:511] Master 127.15.222.126:38087 requested a full tablet report, sending...
I20260809 03:30:37.863991 28628 ts_manager.cc:194] Registered new tserver with Master: b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:37.864600 28628 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:38049
I20260809 03:30:37.865701 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:30:37.870852 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.864407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:40787 (local address 127.15.222.126:38087)
0809 03:30:37.864520 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.864525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.864707 (+ 182us) server_negotiation.cc:407] Connection header received
0809 03:30:37.864856 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.864859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.864899 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.864975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.865367 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.866056 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.866972 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.867241 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.867337 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.868234 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.868237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.868240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.868264 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.869426 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.869799 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.869802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.869804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.869886 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.870126 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.870129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.870131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.870288 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.870369 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.870609 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.870706 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:37.870865 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.864318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.15.222.67:40787)
0809 03:30:37.864642 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.864654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.864767 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.864995 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.864998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.865171 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:37.865264 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.865269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.866183 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.866186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.866850 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.866859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.867333 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.867335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.867398 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.867793 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:37.867807 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:37.868107 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:37.869520 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.869524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.869526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.869701 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.869966 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:37.869969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:37.869971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:37.870049 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:37.870372 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:37.870375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:37.870460 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.870543 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.870675 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:30:37.888826 28706 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:37.883695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.15.222.126:38087)
0809 03:30:37.883812 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.883816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.883828 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:37.883862 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.883864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.883902 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.883967 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.884571 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.885053 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.885833 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.886030 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.886100 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.888716 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58002: BlockingRecv error: recv got EOF from 127.0.0.1:58002 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 03:30:37 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:37.896770 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.890073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.15.222.126:38087)
0809 03:30:37.890173 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.890176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.890333 (+ 157us) server_negotiation.cc:407] Connection header received
0809 03:30:37.890449 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.890451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.890485 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.890545 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:37.891081 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.891555 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.892269 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.892460 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.892529 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.894460 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:37.894463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:37.894465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:37.894488 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:37.895668 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.895981 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.895984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.895985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.896048 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:37.896217 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:37.896218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:37.896219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:37.896361 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:37.896452 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.896587 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.896665 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:30:37.899243 28628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:37.902168 28628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:37.915666 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.909959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.0.0.1:52230)
0809 03:30:37.911102 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.911115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.911206 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.911649 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.911652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.911669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:37.911776 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.911782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.912933 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.912936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.914189 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.914201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915132 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.915136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.915250 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.915291 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.915412 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.915526 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260809 03:30:37.915690 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.910535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.15.222.65:33761)
0809 03:30:37.910914 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.910929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.911205 (+ 276us) server_negotiation.cc:407] Connection header received
0809 03:30:37.911315 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.911321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.911510 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.911654 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:37.911876 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.912778 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.914325 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915018 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.915121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.915280 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.915451 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.915547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":75,"threads_started":1}
I20260809 03:30:37.915900 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.909869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:37705 (local address 127.0.0.1:39512)
0809 03:30:37.910275 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.910322 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.910442 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.911269 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.911276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.911302 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:37.911431 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.911438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.913229 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.913232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.914522 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.914534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915546 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.915549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.915636 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.915658 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.915706 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.915771 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":63,"thread_start_us":61,"threads_started":1}
I20260809 03:30:37.916167 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.910784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:46797 (local address 127.0.0.1:60894)
0809 03:30:37.911563 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:37.911576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:37.911654 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:37.912118 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:37.912122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:37.912140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:37.912226 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.912231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.913538 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.913542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.914725 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:37.914733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915786 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.915789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:37.915876 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.915912 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 03:30:37.915979 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:30:37.916051 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":39,"thread_start_us":83,"threads_started":1}
I20260809 03:30:37.916584 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.909978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.15.222.66:37705)
0809 03:30:37.910746 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.910758 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.910782 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:30:37.910867 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.910872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.911092 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.911587 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:37.911724 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.913087 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.914645 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915415 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.916144 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.916295 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.916368 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.916453 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":110,"threads_started":1}
I20260809 03:30:37.916600 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:37.911175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.15.222.67:46797)
0809 03:30:37.911524 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:37.911534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:37.911737 (+ 203us) server_negotiation.cc:407] Connection header received
0809 03:30:37.911812 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:37.911817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:37.911986 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:37.912124 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:37.912334 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.913424 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.914825 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:37.915482 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:37.915573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:37.915720 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:37.916365 (+ 645us) server_negotiation.cc:299] Negotiation successful
0809 03:30:37.916464 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":119,"threads_started":1}
I20260809 03:30:37.918514 28850 tablet_service.cc:1467] Processing CreateTablet for tablet 856ee0a995854bd789e539160fd2afe1 (DEFAULT_TABLE table=test-table [id=f1748cfa4a164922bc0d25d26f152581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:37.919404 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856ee0a995854bd789e539160fd2afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:37.919313 29306 tablet_service.cc:1467] Processing CreateTablet for tablet 856ee0a995854bd789e539160fd2afe1 (DEFAULT_TABLE table=test-table [id=f1748cfa4a164922bc0d25d26f152581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:37.919755 29075 tablet_service.cc:1467] Processing CreateTablet for tablet 856ee0a995854bd789e539160fd2afe1 (DEFAULT_TABLE table=test-table [id=f1748cfa4a164922bc0d25d26f152581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:37.920359 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856ee0a995854bd789e539160fd2afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:37.920701 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856ee0a995854bd789e539160fd2afe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:37.926162 29475 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Bootstrap starting.
I20260809 03:30:37.927881 29476 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Bootstrap starting.
I20260809 03:30:37.928227 29475 tablet_bootstrap.cc:654] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:37.928776 29478 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Bootstrap starting.
I20260809 03:30:37.929144 29475 log.cc:824] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:37.930068 29476 tablet_bootstrap.cc:654] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:37.930594 29478 tablet_bootstrap.cc:654] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:37.930883 29476 log.cc:824] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:37.931262 29478 log.cc:824] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:37.931397 29475 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: No bootstrap required, opened a new log
I20260809 03:30:37.931576 29475 ts_tablet_manager.cc:1397] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:30:37.932780 29476 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: No bootstrap required, opened a new log
I20260809 03:30:37.932832 29478 tablet_bootstrap.cc:492] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: No bootstrap required, opened a new log
I20260809 03:30:37.932965 29476 ts_tablet_manager.cc:1397] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:30:37.933000 29478 ts_tablet_manager.cc:1397] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 03:30:37.935542 29475 raft_consensus.cc:348] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.935863 29475 raft_consensus.cc:374] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:37.935933 29475 raft_consensus.cc:729] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3d706b01614f74af26febc495994ca, State: Initialized, Role: FOLLOWER
I20260809 03:30:37.936046 29478 raft_consensus.cc:348] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.936288 29478 raft_consensus.cc:374] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:37.936372 29478 raft_consensus.cc:729] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Initialized, Role: FOLLOWER
I20260809 03:30:37.936367 29475 consensus_queue.cc:260] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.936617 29476 raft_consensus.cc:348] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.936774 29478 consensus_queue.cc:260] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.936906 29476 raft_consensus.cc:374] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:37.937029 29476 raft_consensus.cc:729] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Initialized, Role: FOLLOWER
I20260809 03:30:37.937117 28957 heartbeater.cc:503] Master 127.15.222.126:38087 was elected leader, sending a full tablet report...
I20260809 03:30:37.937455 29475 ts_tablet_manager.cc:1428] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:30:37.937490 29476 consensus_queue.cc:260] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:37.937880 29478 ts_tablet_manager.cc:1428] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:30:37.938289 29185 heartbeater.cc:503] Master 127.15.222.126:38087 was elected leader, sending a full tablet report...
I20260809 03:30:37.938289 29413 heartbeater.cc:503] Master 127.15.222.126:38087 was elected leader, sending a full tablet report...
I20260809 03:30:37.938522 29476 ts_tablet_manager.cc:1428] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260809 03:30:37.952316 28958 tablet.cc:2369] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:38.021800 29484 raft_consensus.cc:482] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.022004 29484 raft_consensus.cc:504] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.023123 29484 leader_election.cc:288] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.026852 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.023434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:56839 (local address 127.15.222.66:37705)
0809 03:30:38.023559 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.023564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.023715 (+ 151us) server_negotiation.cc:407] Connection header received
0809 03:30:38.023822 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.023825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.023871 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.023950 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.024223 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.024743 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.025908 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.026277 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.026382 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.026432 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.026672 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.026733 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:30:38.026856 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.023478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:57735 (local address 127.15.222.67:46797)
0809 03:30:38.023603 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.023607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.023799 (+ 192us) server_negotiation.cc:407] Connection header received
0809 03:30:38.023918 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.023920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.023969 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.024040 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.024267 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.024747 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.025860 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.026262 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.026361 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.026415 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.026662 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.026733 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:30:38.026878 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.023348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:46797 (local address 127.15.222.65:57735)
0809 03:30:38.023709 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.023723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.023822 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.024062 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.024065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.024081 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.024165 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.024171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.024848 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.024851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.025747 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.025756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.026374 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.026378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.026536 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.026569 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.026629 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.026709 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":64,"thread_start_us":62,"threads_started":1}
I20260809 03:30:38.026886 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.023307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:37705 (local address 127.15.222.65:56839)
0809 03:30:38.023634 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.023648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.023743 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.023967 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.023970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.023986 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.024117 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.024124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.024844 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.024847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.025775 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.025787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.026382 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.026386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.026539 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.026576 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.026643 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.026733 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":125,"thread_start_us":108,"threads_started":1}
I20260809 03:30:38.027554 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856ee0a995854bd789e539160fd2afe1" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89" is_pre_election: true
I20260809 03:30:38.027545 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856ee0a995854bd789e539160fd2afe1" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f" is_pre_election: true
I20260809 03:30:38.027843 29337 raft_consensus.cc:2454] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.027843 29105 raft_consensus.cc:2454] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.028311 28769 leader_election.cc:302] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23f891195a24a1aa460536b0a56da2f, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.028574 29484 raft_consensus.cc:2781] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:38.028651 29484 raft_consensus.cc:482] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:38.028700 29484 raft_consensus.cc:3037] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.029673 29484 raft_consensus.cc:504] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.029997 29484 leader_election.cc:288] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Requested vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.030293 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856ee0a995854bd789e539160fd2afe1" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f"
I20260809 03:30:38.030300 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856ee0a995854bd789e539160fd2afe1" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89"
I20260809 03:30:38.030405 29105 raft_consensus.cc:3037] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.030403 29337 raft_consensus.cc:3037] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.031603 29105 raft_consensus.cc:2454] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.031625 29337 raft_consensus.cc:2454] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.031873 28765 leader_election.cc:302] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.032038 29484 raft_consensus.cc:2781] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:38.032200 29484 raft_consensus.cc:686] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 1 LEADER]: Becoming Leader. State: Replica: eb3d706b01614f74af26febc495994ca, State: Running, Role: LEADER
I20260809 03:30:38.032498 29484 consensus_queue.cc:237] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.035343 28630 catalog_manager.cc:5495] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3d706b01614f74af26febc495994ca (127.15.222.65). New cstate: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:38.068652 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.065371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.15.222.65:33761)
0809 03:30:38.065504 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.065509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.065524 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:38.065639 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.065641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.065695 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.065800 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.066035 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.066519 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.067317 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.067529 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.067612 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.067859 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.068282 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.068381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.068470 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.068523 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:30:38.074353 29337 raft_consensus.cc:1261] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.074874 29105 raft_consensus.cc:1261] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.075022 29489 consensus_queue.cc:1035] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.075404 29484 consensus_queue.cc:1035] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.091944 28628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.092090 28628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.094304 28628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Sending DeleteTablet for 3 replicas of tablet 856ee0a995854bd789e539160fd2afe1
I20260809 03:30:38.094976 29075 tablet_service.cc:1514] Processing DeleteTablet for tablet 856ee0a995854bd789e539160fd2afe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260809 03:30:38.095049 28850 tablet_service.cc:1514] Processing DeleteTablet for tablet 856ee0a995854bd789e539160fd2afe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:30:38.095407 29507 tablet_replica.cc:330] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: stopping tablet replica
I20260809 03:30:38.095510 29508 tablet_replica.cc:330] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: stopping tablet replica
I20260809 03:30:38.095796 29507 raft_consensus.cc:2229] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.095794 29508 raft_consensus.cc:2229] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:38.095747 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet 856ee0a995854bd789e539160fd2afe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:60894
I20260809 03:30:38.096149 29508 raft_consensus.cc:2258] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.096195 29507 raft_consensus.cc:2258] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.096940 28628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:38.097324 28628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:38.097468 29508 ts_tablet_manager.cc:1905] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.097855 29507 ts_tablet_manager.cc:1905] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.101240 29507 ts_tablet_manager.cc:1918] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.101529 29508 ts_tablet_manager.cc:1918] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.101611 29508 log.cc:1196] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/856ee0a995854bd789e539160fd2afe1
I20260809 03:30:38.101950 29508 ts_tablet_manager.cc:1939] T 856ee0a995854bd789e539160fd2afe1 P eb3d706b01614f74af26febc495994ca: Deleting consensus metadata
I20260809 03:30:38.102095 28850 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b (DEFAULT_TABLE table=test-table [id=a73206ceea44463aa3cc6480badebf71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.102334 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ebfe92ddb4b60b151013e41b57d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.103938 29475 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Bootstrap starting.
I20260809 03:30:38.104106 29075 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b (DEFAULT_TABLE table=test-table [id=a73206ceea44463aa3cc6480badebf71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.104337 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ebfe92ddb4b60b151013e41b57d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.104851 29475 tablet_bootstrap.cc:654] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.106024 28604 catalog_manager.cc:4841] TS eb3d706b01614f74af26febc495994ca (127.15.222.65:33761): tablet 856ee0a995854bd789e539160fd2afe1 (table test-table [id=f1748cfa4a164922bc0d25d26f152581]) successfully deleted
I20260809 03:30:38.106089 29475 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: No bootstrap required, opened a new log
I20260809 03:30:38.106164 29475 ts_tablet_manager.cc:1397] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:38.101334 29507 log.cc:1196] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/856ee0a995854bd789e539160fd2afe1
I20260809 03:30:38.106576 29475 raft_consensus.cc:348] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.106691 29475 raft_consensus.cc:374] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.106724 29478 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Bootstrap starting.
I20260809 03:30:38.106732 29475 raft_consensus.cc:729] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3d706b01614f74af26febc495994ca, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.106654 29507 ts_tablet_manager.cc:1939] T 856ee0a995854bd789e539160fd2afe1 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting consensus metadata
I20260809 03:30:38.106842 29475 consensus_queue.cc:260] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.107116 29475 ts_tablet_manager.cc:1428] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:38.107554 29478 tablet_bootstrap.cc:654] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.108086 29509 tablet_replica.cc:330] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: stopping tablet replica
I20260809 03:30:38.108183 29306 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b (DEFAULT_TABLE table=test-table [id=a73206ceea44463aa3cc6480badebf71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.108435 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ebfe92ddb4b60b151013e41b57d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.108490 29509 raft_consensus.cc:2229] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.108769 29509 raft_consensus.cc:2258] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.110059 29509 ts_tablet_manager.cc:1905] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.110507 29476 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Bootstrap starting.
I20260809 03:30:38.111483 29476 tablet_bootstrap.cc:654] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.112270 28601 catalog_manager.cc:4841] TS 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705): tablet 856ee0a995854bd789e539160fd2afe1 (table test-table [id=f1748cfa4a164922bc0d25d26f152581]) successfully deleted
I20260809 03:30:38.112393 29509 ts_tablet_manager.cc:1918] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.112483 29509 log.cc:1196] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/856ee0a995854bd789e539160fd2afe1
I20260809 03:30:38.112761 29509 ts_tablet_manager.cc:1939] T 856ee0a995854bd789e539160fd2afe1 P b23f891195a24a1aa460536b0a56da2f: Deleting consensus metadata
I20260809 03:30:38.113020 29478 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: No bootstrap required, opened a new log
I20260809 03:30:38.113095 29478 ts_tablet_manager.cc:1397] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260809 03:30:38.113482 29478 raft_consensus.cc:348] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.113572 29478 raft_consensus.cc:374] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.113601 29478 raft_consensus.cc:729] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.113751 29478 consensus_queue.cc:260] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.113986 29478 ts_tablet_manager.cc:1428] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.114271 28604 catalog_manager.cc:4841] TS b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797): tablet 856ee0a995854bd789e539160fd2afe1 (table test-table [id=f1748cfa4a164922bc0d25d26f152581]) successfully deleted
I20260809 03:30:38.115474 29476 tablet_bootstrap.cc:492] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: No bootstrap required, opened a new log
I20260809 03:30:38.115561 29476 ts_tablet_manager.cc:1397] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260809 03:30:38.115998 29476 raft_consensus.cc:348] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.116101 29476 raft_consensus.cc:374] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.116134 29476 raft_consensus.cc:729] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.116240 29476 consensus_queue.cc:260] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.116487 29476 ts_tablet_manager.cc:1428] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.138100 29486 raft_consensus.cc:482] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.138209 29486 raft_consensus.cc:504] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.139256 29486 leader_election.cc:288] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb3d706b01614f74af26febc495994ca (127.15.222.65:33761), 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705)
I20260809 03:30:38.143746 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.139490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:37705 (local address 127.15.222.67:39311)
0809 03:30:38.139833 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.139846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.139946 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.140401 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.140403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.140418 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.140521 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.140526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.141389 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.141393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.142513 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.142523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.143313 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.143315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.143404 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.143461 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.143529 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.143594 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":7,"thread_start_us":75,"threads_started":1}
I20260809 03:30:38.143762 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.139752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:39311 (local address 127.15.222.66:37705)
0809 03:30:38.139851 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.139855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.139911 (+ 56us) server_negotiation.cc:407] Connection header received
0809 03:30:38.140026 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.140029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.140065 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.140131 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.140633 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.141293 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.142650 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.143110 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.143182 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.143222 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.143567 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.143643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:30:38.143992 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.139575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:32835 (local address 127.15.222.65:33761)
0809 03:30:38.139710 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.139715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.139966 (+ 251us) server_negotiation.cc:407] Connection header received
0809 03:30:38.140017 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.140020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.140066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.140133 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.140386 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.141099 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.142683 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.143285 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.143408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.143455 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.143806 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.143890 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:38.143754 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.139450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.15.222.67:32835)
0809 03:30:38.139771 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.139786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.139873 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.140150 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.140153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.140169 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.140282 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.140289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.141196 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.141199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.142546 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.142557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.143402 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.143406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.143539 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.143557 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.143614 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.143667 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":68}
I20260809 03:30:38.144460 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ebfe92ddb4b60b151013e41b57d3b" candidate_uuid: "b23f891195a24a1aa460536b0a56da2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89" is_pre_election: true
I20260809 03:30:38.144601 29105 raft_consensus.cc:2454] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23f891195a24a1aa460536b0a56da2f in term 0.
I20260809 03:30:38.144685 28878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ebfe92ddb4b60b151013e41b57d3b" candidate_uuid: "b23f891195a24a1aa460536b0a56da2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3d706b01614f74af26febc495994ca" is_pre_election: true
I20260809 03:30:38.144930 28878 raft_consensus.cc:2454] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23f891195a24a1aa460536b0a56da2f in term 0.
I20260809 03:30:38.144963 29223 leader_election.cc:302] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, b23f891195a24a1aa460536b0a56da2f; no voters:
I20260809 03:30:38.145200 29486 raft_consensus.cc:2781] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:38.145254 29486 raft_consensus.cc:482] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:38.145290 29486 raft_consensus.cc:3037] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.146260 29486 raft_consensus.cc:504] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.146564 29486 leader_election.cc:288] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [CANDIDATE]: Term 1 election: Requested vote from peers eb3d706b01614f74af26febc495994ca (127.15.222.65:33761), 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705)
I20260809 03:30:38.146847 28878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ebfe92ddb4b60b151013e41b57d3b" candidate_uuid: "b23f891195a24a1aa460536b0a56da2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3d706b01614f74af26febc495994ca"
I20260809 03:30:38.146867 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ebfe92ddb4b60b151013e41b57d3b" candidate_uuid: "b23f891195a24a1aa460536b0a56da2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89"
I20260809 03:30:38.146952 28878 raft_consensus.cc:3037] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.146970 29105 raft_consensus.cc:3037] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.147812 28878 raft_consensus.cc:2454] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23f891195a24a1aa460536b0a56da2f in term 1.
I20260809 03:30:38.148005 29105 raft_consensus.cc:2454] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23f891195a24a1aa460536b0a56da2f in term 1.
I20260809 03:30:38.148072 29225 leader_election.cc:302] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23f891195a24a1aa460536b0a56da2f, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.148324 29486 raft_consensus.cc:2781] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:38.148527 29486 raft_consensus.cc:686] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 1 LEADER]: Becoming Leader. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Running, Role: LEADER
I20260809 03:30:38.148757 29486 consensus_queue.cc:237] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.150889 28628 catalog_manager.cc:5495] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f reported cstate change: term changed from 0 to 1, leader changed from <none> to b23f891195a24a1aa460536b0a56da2f (127.15.222.67). New cstate: current_term: 1 leader_uuid: "b23f891195a24a1aa460536b0a56da2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } health_report { overall_health: HEALTHY } } }
W20260809 03:30:38.158736 29187 tablet.cc:2369] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 03:30:38 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246237, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.126@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:38.188141 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.179320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38087 (local address 127.0.0.1:58016)
0809 03:30:38.179611 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.179623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.179722 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.179980 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.179983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.180446 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:38.180554 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.180562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.181602 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.181605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.182284 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.182293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.182843 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.182845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.182901 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.183422 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:38.183434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:38.184981 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:38.186774 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:38.186783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:38.186787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:38.187012 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:38.187259 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:38.187260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:38.187262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:38.187330 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:38.187730 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:38.187734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:38.187827 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.187914 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.188020 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260809 03:30:38.188412 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.179368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.15.222.126:38087)
0809 03:30:38.179482 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.179487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.179700 (+ 213us) server_negotiation.cc:407] Connection header received
0809 03:30:38.179823 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.179826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.179878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.179955 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:38.180680 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.181495 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.182404 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.182751 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.182848 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.185112 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:38.185116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:38.185119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:38.185145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:38.186677 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:38.187095 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:38.187098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:38.187100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:38.187185 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:38.187409 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:38.187412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:38.187414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:38.187640 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:38.187742 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.187987 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.188100 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:30:38.192317 28628 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58016:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 03:30:38.193105 28628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:38.198052 29306 tablet_service.cc:1467] Processing CreateTablet for tablet b973b1b7949b4a699adedce4635600b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45f252e6ded24865a34ff2ee58c23b75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:38.198087 28850 tablet_service.cc:1467] Processing CreateTablet for tablet b973b1b7949b4a699adedce4635600b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45f252e6ded24865a34ff2ee58c23b75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:38.198210 29075 tablet_service.cc:1467] Processing CreateTablet for tablet b973b1b7949b4a699adedce4635600b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45f252e6ded24865a34ff2ee58c23b75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 03:30:38.198493 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b973b1b7949b4a699adedce4635600b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.198495 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b973b1b7949b4a699adedce4635600b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.198534 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b973b1b7949b4a699adedce4635600b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.201346 29478 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: Bootstrap starting.
I20260809 03:30:38.201479 29476 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: Bootstrap starting.
I20260809 03:30:38.202363 29476 tablet_bootstrap.cc:654] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.202515 29478 tablet_bootstrap.cc:654] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.203658 29476 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: No bootstrap required, opened a new log
I20260809 03:30:38.203740 29476 ts_tablet_manager.cc:1397] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:38.203809 29478 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: No bootstrap required, opened a new log
I20260809 03:30:38.203881 29478 ts_tablet_manager.cc:1397] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 03:30:38.204289 29478 raft_consensus.cc:348] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.204406 29478 raft_consensus.cc:374] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.204443 29478 raft_consensus.cc:729] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.204411 29476 raft_consensus.cc:348] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.204512 29476 raft_consensus.cc:374] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.204541 29476 raft_consensus.cc:729] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.204566 29478 consensus_queue.cc:260] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.204639 29476 consensus_queue.cc:260] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.204882 29476 ts_tablet_manager.cc:1428] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:38.204835 29485 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.204991 29485 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.204995 29486 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.205093 29486 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.204838 29478 ts_tablet_manager.cc:1428] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:38.206908 29475 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: Bootstrap starting.
I20260809 03:30:38.207816 29475 tablet_bootstrap.cc:654] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.209141 29475 tablet_bootstrap.cc:492] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: No bootstrap required, opened a new log
I20260809 03:30:38.209211 29475 ts_tablet_manager.cc:1397] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:38.209547 29475 raft_consensus.cc:348] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.209635 29475 raft_consensus.cc:374] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.209653 29475 raft_consensus.cc:729] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3d706b01614f74af26febc495994ca, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.209771 29475 consensus_queue.cc:260] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.210049 29475 ts_tablet_manager.cc:1428] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 03:30:38.210004 29502 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.210134 29502 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.339881 29502 raft_consensus.cc:482] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.340019 29502 raft_consensus.cc:504] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.340438 29502 leader_election.cc:288] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.340809 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b973b1b7949b4a699adedce4635600b6" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89" is_pre_election: true
I20260809 03:30:38.340816 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b973b1b7949b4a699adedce4635600b6" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f" is_pre_election: true
I20260809 03:30:38.340957 29337 raft_consensus.cc:2454] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.340963 29105 raft_consensus.cc:2454] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.341251 28765 leader_election.cc:302] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.341416 29502 raft_consensus.cc:2781] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:38.341485 29502 raft_consensus.cc:482] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:38.341533 29502 raft_consensus.cc:3037] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.342530 29502 raft_consensus.cc:504] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.342816 29502 leader_election.cc:288] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Requested vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.343143 29105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b973b1b7949b4a699adedce4635600b6" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89"
I20260809 03:30:38.343160 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b973b1b7949b4a699adedce4635600b6" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f"
I20260809 03:30:38.343240 29105 raft_consensus.cc:3037] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.343257 29337 raft_consensus.cc:3037] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.344061 29105 raft_consensus.cc:2454] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.344206 29337 raft_consensus.cc:2454] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.344312 28765 leader_election.cc:302] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.344482 29502 raft_consensus.cc:2781] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:38.344564 29502 raft_consensus.cc:686] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [term 1 LEADER]: Becoming Leader. State: Replica: eb3d706b01614f74af26febc495994ca, State: Running, Role: LEADER
I20260809 03:30:38.344703 29502 consensus_queue.cc:237] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.345371 29489 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: TxnStatusTablet state changed. Reason: New leader eb3d706b01614f74af26febc495994ca. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.345492 29489 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:38.345816 29534 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 03:30:38.346014 29534 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 03:30:38.346400 28630 catalog_manager.cc:5495] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3d706b01614f74af26febc495994ca (127.15.222.65). New cstate: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } health_report { overall_health: UNKNOWN } } }
W20260809 03:30:38.359906 29415 tablet.cc:2369] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:30:38.363229 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.359527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.15.222.65:33761)
0809 03:30:38.359643 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.359647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.359677 (+ 30us) server_negotiation.cc:407] Connection header received
0809 03:30:38.359883 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.359886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.359941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.360014 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.360378 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.360877 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.362103 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.362369 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.362475 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.362778 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.362874 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.362958 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.363058 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.363106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:30:38.363267 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.359472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.0.0.1:52240)
0809 03:30:38.359618 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.359633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.359748 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.360142 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.360145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.360152 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:38.360256 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.360262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.360976 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.360979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.361980 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.361991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.362486 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.362490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.362583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.362640 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.362960 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.362971 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.363029 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.363113 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:30:38.365757 29337 raft_consensus.cc:1261] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.365782 29105 raft_consensus.cc:1261] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.366120 29502 consensus_queue.cc:1035] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.366259 29489 consensus_queue.cc:1035] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.367467 29485 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: TxnStatusTablet state changed. Reason: New leader eb3d706b01614f74af26febc495994ca. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.367488 29486 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: TxnStatusTablet state changed. Reason: New leader eb3d706b01614f74af26febc495994ca. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.367583 29485 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.367583 29486 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.368734 29502 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.368844 29502 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:38.369266 29494 mvcc.cc:205] Tried to move back new op lower bound from 7316464592342212608 to 7316464592261177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464592342212608}]
I20260809 03:30:38.369472 29489 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.369560 29489 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P eb3d706b01614f74af26febc495994ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 03:30:38.369547 29486 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.369629 29486 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P b23f891195a24a1aa460536b0a56da2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.369639 29498 mvcc.cc:205] Tried to move back new op lower bound from 7316464592342212608 to 7316464592261177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464592342212608}]
I20260809 03:30:38.372879 29497 mvcc.cc:205] Tried to move back new op lower bound from 7316464592342212608 to 7316464592261177344. Current Snapshot: MvccSnapshot[applied={T|T < 7316464592342212608}]
I20260809 03:30:38.373039 29485 tablet_replica.cc:439] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } } }
I20260809 03:30:38.373122 29485 tablet_replica.cc:442] T b973b1b7949b4a699adedce4635600b6 P 91e7b9fd9bd64c459a270c1f30362a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 03:30:38.377517 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.373871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.15.222.67:46797)
0809 03:30:38.373978 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.373982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.373995 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:38.374063 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.374066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.374123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.374223 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.374477 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.375213 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.376014 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.376373 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.376482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.376661 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.377180 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.377310 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.377340 (+ 30us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.377394 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:30:38.387281 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.384011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.15.222.67:47561)
0809 03:30:38.384284 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.384297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.384378 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.384582 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.384584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.384589 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:38.384671 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.384674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.385562 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.385564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.386201 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.386209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.386733 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.386736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.386795 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.386833 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.387074 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.387078 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.387116 (+ 38us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.387170 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260809 03:30:38.387310 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.384101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:47561 (local address 127.15.222.65:33761)
0809 03:30:38.384211 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.384215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.384359 (+ 144us) server_negotiation.cc:407] Connection header received
0809 03:30:38.384453 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.384456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.384504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.384580 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.384736 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.385478 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.386296 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.386644 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.386741 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.386934 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.387006 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.387064 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.387127 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.387179 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:38.393975 29105 raft_consensus.cc:1261] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Refusing update from remote peer b23f891195a24a1aa460536b0a56da2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.394165 28878 raft_consensus.cc:1261] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Refusing update from remote peer b23f891195a24a1aa460536b0a56da2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.394387 29519 consensus_queue.cc:1035] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.394634 29486 consensus_queue.cc:1035] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.415230 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.411604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:56477 (local address 127.15.222.67:46797)
0809 03:30:38.411708 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.411712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.411876 (+ 164us) server_negotiation.cc:407] Connection header received
0809 03:30:38.411976 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.411979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.412046 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.412117 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.412477 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.413128 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.414300 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.414524 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.414608 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.414853 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.414921 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.414978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.415061 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.415106 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:30:38.415246 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.411521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:46797 (local address 127.15.222.65:56477)
0809 03:30:38.411817 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.411830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.411912 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.412225 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.412228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.412236 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:38.412344 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.412350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.413453 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.413456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.414197 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.414207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.414618 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.414621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.414699 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.414738 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.414990 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.414996 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.415053 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.415126 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260809 03:30:38.437412 28630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.437574 28630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.439957 28630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Sending DeleteTablet for 3 replicas of tablet 4a6ebfe92ddb4b60b151013e41b57d3b
I20260809 03:30:38.440622 29075 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260809 03:30:38.440888 29507 tablet_replica.cc:330] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: stopping tablet replica
I20260809 03:30:38.441013 29507 raft_consensus.cc:2229] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.441116 29507 raft_consensus.cc:2258] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.442124 29507 ts_tablet_manager.cc:1905] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.445014 29507 ts_tablet_manager.cc:1918] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:38.445096 29507 log.cc:1196] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/4a6ebfe92ddb4b60b151013e41b57d3b
I20260809 03:30:38.445343 29507 ts_tablet_manager.cc:1939] T 4a6ebfe92ddb4b60b151013e41b57d3b P 91e7b9fd9bd64c459a270c1f30362a89: Deleting consensus metadata
I20260809 03:30:38.445444 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:60894
I20260809 03:30:38.445614 29509 tablet_replica.cc:330] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: stopping tablet replica
I20260809 03:30:38.445748 29509 raft_consensus.cc:2229] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:38.446020 29509 raft_consensus.cc:2258] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.448361 28850 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ebfe92ddb4b60b151013e41b57d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:30:38.448537 29508 tablet_replica.cc:330] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: stopping tablet replica
I20260809 03:30:38.448632 29508 raft_consensus.cc:2229] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.448733 29508 raft_consensus.cc:2258] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.449486 29508 ts_tablet_manager.cc:1905] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.450254 29509 ts_tablet_manager.cc:1905] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.452853 29509 ts_tablet_manager.cc:1918] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:38.452919 29509 log.cc:1196] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/4a6ebfe92ddb4b60b151013e41b57d3b
I20260809 03:30:38.453150 29509 ts_tablet_manager.cc:1939] T 4a6ebfe92ddb4b60b151013e41b57d3b P b23f891195a24a1aa460536b0a56da2f: Deleting consensus metadata
I20260809 03:30:38.453807 28604 catalog_manager.cc:4841] TS b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797): tablet 4a6ebfe92ddb4b60b151013e41b57d3b (table test-table [id=a73206ceea44463aa3cc6480badebf71]) successfully deleted
I20260809 03:30:38.456367 29508 ts_tablet_manager.cc:1918] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:38.456457 29508 log.cc:1196] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/4a6ebfe92ddb4b60b151013e41b57d3b
I20260809 03:30:38.456691 29508 ts_tablet_manager.cc:1939] T 4a6ebfe92ddb4b60b151013e41b57d3b P eb3d706b01614f74af26febc495994ca: Deleting consensus metadata
I20260809 03:30:38.457365 28601 catalog_manager.cc:4841] TS 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705): tablet 4a6ebfe92ddb4b60b151013e41b57d3b (table test-table [id=a73206ceea44463aa3cc6480badebf71]) successfully deleted
I20260809 03:30:38.457645 28604 catalog_manager.cc:4841] TS eb3d706b01614f74af26febc495994ca (127.15.222.65:33761): tablet 4a6ebfe92ddb4b60b151013e41b57d3b (table test-table [id=a73206ceea44463aa3cc6480badebf71]) successfully deleted
Aug 09 03:30:38 dist-test-slave-cvwc krb5kdc[28558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 03:30:38 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:38 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:30:38.492242 28706 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:38.484512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.15.222.126:38087)
0809 03:30:38.484646 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.484650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.484930 (+ 280us) server_negotiation.cc:407] Connection header received
0809 03:30:38.485065 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.485069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.485125 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.485209 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:38.486127 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.486978 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.488148 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.488503 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.488613 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.492117 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58024: BlockingRecv error: recv got EOF from 127.0.0.1:58024 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 03:30:38 dist-test-slave-cvwc krb5kdc[28558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM
I20260809 03:30:38.503861 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.494391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.15.222.126:38087)
0809 03:30:38.494489 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.494493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.494660 (+ 167us) server_negotiation.cc:407] Connection header received
0809 03:30:38.494760 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.494769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.494815 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.494892 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:38.495547 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.496317 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.497405 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.497751 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.497849 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.500471 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:38.500475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:38.500478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:38.500506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:38.502095 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:38.502557 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:38.502561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:38.502563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:38.502652 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:38.502887 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:38.502890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:38.502892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:38.503122 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:38.503247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.503538 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.503674 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:30:38.506659 28630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:30:38.507112 28630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:38.512553 28850 tablet_service.cc:1467] Processing CreateTablet for tablet b85803414a0e404eaabd514238d8cd83 (DEFAULT_TABLE table=test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.512884 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85803414a0e404eaabd514238d8cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.514024 29075 tablet_service.cc:1467] Processing CreateTablet for tablet b85803414a0e404eaabd514238d8cd83 (DEFAULT_TABLE table=test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.514297 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85803414a0e404eaabd514238d8cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.515462 29475 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Bootstrap starting.
I20260809 03:30:38.516291 29475 tablet_bootstrap.cc:654] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.516719 29478 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Bootstrap starting.
I20260809 03:30:38.518378 29475 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: No bootstrap required, opened a new log
I20260809 03:30:38.518469 29475 ts_tablet_manager.cc:1397] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260809 03:30:38.518621 29478 tablet_bootstrap.cc:654] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.518903 29475 raft_consensus.cc:348] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.519022 29475 raft_consensus.cc:374] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.519061 29475 raft_consensus.cc:729] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3d706b01614f74af26febc495994ca, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.517779 29306 tablet_service.cc:1467] Processing CreateTablet for tablet b85803414a0e404eaabd514238d8cd83 (DEFAULT_TABLE table=test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.519183 29475 consensus_queue.cc:260] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.519423 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b85803414a0e404eaabd514238d8cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.519440 29475 ts_tablet_manager.cc:1428] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:38.521595 29476 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Bootstrap starting.
I20260809 03:30:38.522444 29478 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: No bootstrap required, opened a new log
I20260809 03:30:38.522534 29478 ts_tablet_manager.cc:1397] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 03:30:38.522768 29476 tablet_bootstrap.cc:654] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.523010 29478 raft_consensus.cc:348] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.523120 29478 raft_consensus.cc:374] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.523154 29478 raft_consensus.cc:729] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.523290 29478 consensus_queue.cc:260] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.523573 29478 ts_tablet_manager.cc:1428] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.524242 29476 tablet_bootstrap.cc:492] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: No bootstrap required, opened a new log
I20260809 03:30:38.524328 29476 ts_tablet_manager.cc:1397] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:30:38.524724 29476 raft_consensus.cc:348] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.524816 29476 raft_consensus.cc:374] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.524845 29476 raft_consensus.cc:729] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.524967 29476 consensus_queue.cc:260] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.525655 29476 ts_tablet_manager.cc:1428] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.668749 29484 raft_consensus.cc:482] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.668887 29484 raft_consensus.cc:504] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.669281 29484 leader_election.cc:288] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.669643 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85803414a0e404eaabd514238d8cd83" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f" is_pre_election: true
I20260809 03:30:38.669668 29104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85803414a0e404eaabd514238d8cd83" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89" is_pre_election: true
I20260809 03:30:38.669816 29337 raft_consensus.cc:2454] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.669837 29104 raft_consensus.cc:2454] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 0.
I20260809 03:30:38.670089 28769 leader_election.cc:302] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23f891195a24a1aa460536b0a56da2f, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.670264 29484 raft_consensus.cc:2781] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:38.670327 29484 raft_consensus.cc:482] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:38.670361 29484 raft_consensus.cc:3037] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.671355 29484 raft_consensus.cc:504] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.671663 29484 leader_election.cc:288] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Requested vote from peers 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.671991 29104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85803414a0e404eaabd514238d8cd83" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e7b9fd9bd64c459a270c1f30362a89"
I20260809 03:30:38.672016 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85803414a0e404eaabd514238d8cd83" candidate_uuid: "eb3d706b01614f74af26febc495994ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f"
I20260809 03:30:38.672122 29337 raft_consensus.cc:3037] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.672122 29104 raft_consensus.cc:3037] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.672967 29104 raft_consensus.cc:2454] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.673209 29337 raft_consensus.cc:2454] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3d706b01614f74af26febc495994ca in term 1.
I20260809 03:30:38.673210 28765 leader_election.cc:302] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.673358 29484 raft_consensus.cc:2781] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:38.673448 29484 raft_consensus.cc:686] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 1 LEADER]: Becoming Leader. State: Replica: eb3d706b01614f74af26febc495994ca, State: Running, Role: LEADER
I20260809 03:30:38.673578 29484 consensus_queue.cc:237] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.675168 28630 catalog_manager.cc:5495] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3d706b01614f74af26febc495994ca (127.15.222.65). New cstate: current_term: 1 leader_uuid: "eb3d706b01614f74af26febc495994ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:38.704569 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.701795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.15.222.65:33761)
0809 03:30:38.701909 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.701913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.701927 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:38.702009 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.702011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.702053 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.702112 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:38.702345 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.702840 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.703724 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.703933 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.704004 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.704246 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:38.704311 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:38.704352 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.704422 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.704462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:30:38.706343 29104 raft_consensus.cc:1261] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.706419 29337 raft_consensus.cc:1261] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Refusing update from remote peer eb3d706b01614f74af26febc495994ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:38.706723 29484 consensus_queue.cc:1035] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.706866 29489 consensus_queue.cc:1035] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:38.714018 28628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.714134 28628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:38.715848 28628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Sending DeleteTablet for 3 replicas of tablet b85803414a0e404eaabd514238d8cd83
I20260809 03:30:38.716333 28850 tablet_service.cc:1514] Processing DeleteTablet for tablet b85803414a0e404eaabd514238d8cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:30:38.716358 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet b85803414a0e404eaabd514238d8cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:60894
I20260809 03:30:38.716362 29075 tablet_service.cc:1514] Processing DeleteTablet for tablet b85803414a0e404eaabd514238d8cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:38 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260809 03:30:38.716501 29509 tablet_replica.cc:330] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: stopping tablet replica
I20260809 03:30:38.716501 29508 tablet_replica.cc:330] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: stopping tablet replica
I20260809 03:30:38.716524 29507 tablet_replica.cc:330] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: stopping tablet replica
I20260809 03:30:38.716610 29507 raft_consensus.cc:2229] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.716606 29509 raft_consensus.cc:2229] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:38.716607 29508 raft_consensus.cc:2229] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:38.716694 29507 raft_consensus.cc:2258] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.716713 29509 raft_consensus.cc:2258] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.716756 29508 raft_consensus.cc:2258] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:38.717396 29508 ts_tablet_manager.cc:1905] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.717466 29509 ts_tablet_manager.cc:1905] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:38.717422 28628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 03:30:38.717705 29507 ts_tablet_manager.cc:1905] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:30:38.717795 28628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:30:38.719827 29508 ts_tablet_manager.cc:1918] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.719866 29507 ts_tablet_manager.cc:1918] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.719866 29509 ts_tablet_manager.cc:1918] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:30:38.719900 29508 log.cc:1196] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/b85803414a0e404eaabd514238d8cd83
I20260809 03:30:38.719936 29509 log.cc:1196] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/b85803414a0e404eaabd514238d8cd83
I20260809 03:30:38.719936 29507 log.cc:1196] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/b85803414a0e404eaabd514238d8cd83
I20260809 03:30:38.720141 29508 ts_tablet_manager.cc:1939] T b85803414a0e404eaabd514238d8cd83 P eb3d706b01614f74af26febc495994ca: Deleting consensus metadata
I20260809 03:30:38.720188 29509 ts_tablet_manager.cc:1939] T b85803414a0e404eaabd514238d8cd83 P b23f891195a24a1aa460536b0a56da2f: Deleting consensus metadata
I20260809 03:30:38.720188 29507 ts_tablet_manager.cc:1939] T b85803414a0e404eaabd514238d8cd83 P 91e7b9fd9bd64c459a270c1f30362a89: Deleting consensus metadata
I20260809 03:30:38.720728 28604 catalog_manager.cc:4841] TS b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797): tablet b85803414a0e404eaabd514238d8cd83 (table test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]) successfully deleted
I20260809 03:30:38.720794 28601 catalog_manager.cc:4841] TS 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705): tablet b85803414a0e404eaabd514238d8cd83 (table test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]) successfully deleted
I20260809 03:30:38.720885 28604 catalog_manager.cc:4841] TS eb3d706b01614f74af26febc495994ca (127.15.222.65:33761): tablet b85803414a0e404eaabd514238d8cd83 (table test-table [id=dfc0ed44ee4d455f9ca54ca9da47b080]) successfully deleted
I20260809 03:30:38.722993 28850 tablet_service.cc:1467] Processing CreateTablet for tablet 2914ee58a8624639b398c05117b14ecb (DEFAULT_TABLE table=test-table [id=c07fa7f256ab49ad8377613556805ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.723042 29075 tablet_service.cc:1467] Processing CreateTablet for tablet 2914ee58a8624639b398c05117b14ecb (DEFAULT_TABLE table=test-table [id=c07fa7f256ab49ad8377613556805ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.723232 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2914ee58a8624639b398c05117b14ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.723273 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2914ee58a8624639b398c05117b14ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.723237 29306 tablet_service.cc:1467] Processing CreateTablet for tablet 2914ee58a8624639b398c05117b14ecb (DEFAULT_TABLE table=test-table [id=c07fa7f256ab49ad8377613556805ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:30:38.723456 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2914ee58a8624639b398c05117b14ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:38.724839 29475 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: Bootstrap starting.
I20260809 03:30:38.725312 29478 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Bootstrap starting.
I20260809 03:30:38.725471 29476 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Bootstrap starting.
I20260809 03:30:38.725589 29475 tablet_bootstrap.cc:654] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.726128 29478 tablet_bootstrap.cc:654] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.726187 29476 tablet_bootstrap.cc:654] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:38.726531 29475 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: No bootstrap required, opened a new log
I20260809 03:30:38.726603 29475 ts_tablet_manager.cc:1397] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:38.726926 29475 raft_consensus.cc:348] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.727023 29475 raft_consensus.cc:374] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.727052 29475 raft_consensus.cc:729] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3d706b01614f74af26febc495994ca, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.727164 29475 consensus_queue.cc:260] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.727430 29475 ts_tablet_manager.cc:1428] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:30:38.728048 29476 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: No bootstrap required, opened a new log
I20260809 03:30:38.728132 29476 ts_tablet_manager.cc:1397] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 03:30:38.728305 29478 tablet_bootstrap.cc:492] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: No bootstrap required, opened a new log
I20260809 03:30:38.728382 29478 ts_tablet_manager.cc:1397] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 03:30:38.728507 29476 raft_consensus.cc:348] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.728610 29476 raft_consensus.cc:374] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.728649 29476 raft_consensus.cc:729] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f891195a24a1aa460536b0a56da2f, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.728763 29476 consensus_queue.cc:260] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.728785 29478 raft_consensus.cc:348] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.728869 29478 raft_consensus.cc:374] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:38.728907 29478 raft_consensus.cc:729] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Initialized, Role: FOLLOWER
I20260809 03:30:38.729040 29476 ts_tablet_manager.cc:1428] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.729032 29478 consensus_queue.cc:260] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.729319 29478 ts_tablet_manager.cc:1428] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 03:30:38.987879 29485 raft_consensus.cc:482] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:30:38.988047 29485 raft_consensus.cc:504] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.989138 29485 leader_election.cc:288] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb3d706b01614f74af26febc495994ca (127.15.222.65:33761), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.993287 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.989489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40291 (local address 127.15.222.65:33761)
0809 03:30:38.989627 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.989631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.989785 (+ 154us) server_negotiation.cc:407] Connection header received
0809 03:30:38.989985 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.989989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.990043 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.990093 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.990526 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.991155 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992374 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.992772 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992877 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.992935 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.993100 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.993163 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:30:38.993316 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.989369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.15.222.66:40291)
0809 03:30:38.989667 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.989703 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.989841 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.990246 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.990249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.990268 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.990409 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.990416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.991285 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.991288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.992252 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.992263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.992878 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.992992 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.993011 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.993074 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.993142 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":52,"thread_start_us":95,"threads_started":1}
I20260809 03:30:38.993366 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.989773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:46797 (local address 127.15.222.66:47321)
0809 03:30:38.990064 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:38.990076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:38.990162 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:38.990401 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:38.990403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:38.990416 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:30:38.990564 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.990571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.991365 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.991368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.992245 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:38.992255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.992917 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:38.993055 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.993075 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 03:30:38.993138 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:30:38.993206 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260809 03:30:38.993409 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:38.989888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:47321 (local address 127.15.222.67:46797)
0809 03:30:38.990120 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:38.990124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:38.990139 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:38.990257 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:38.990260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:38.990306 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:38.990392 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:30:38.990678 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.991263 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992357 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:38.992793 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:38.992902 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:38.992954 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:38.993205 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 03:30:38.993285 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260809 03:30:38.993732 28878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2914ee58a8624639b398c05117b14ecb" candidate_uuid: "91e7b9fd9bd64c459a270c1f30362a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3d706b01614f74af26febc495994ca" is_pre_election: true
I20260809 03:30:38.993894 28878 raft_consensus.cc:2454] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e7b9fd9bd64c459a270c1f30362a89 in term 0.
I20260809 03:30:38.993916 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2914ee58a8624639b398c05117b14ecb" candidate_uuid: "91e7b9fd9bd64c459a270c1f30362a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f" is_pre_election: true
I20260809 03:30:38.994035 29337 raft_consensus.cc:2454] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e7b9fd9bd64c459a270c1f30362a89 in term 0.
I20260809 03:30:38.994374 28992 leader_election.cc:302] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, eb3d706b01614f74af26febc495994ca; no voters:
I20260809 03:30:38.994603 29485 raft_consensus.cc:2781] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:30:38.994657 29485 raft_consensus.cc:482] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:30:38.994683 29485 raft_consensus.cc:3037] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.995671 29485 raft_consensus.cc:504] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:38.995982 29485 leader_election.cc:288] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [CANDIDATE]: Term 1 election: Requested vote from peers eb3d706b01614f74af26febc495994ca (127.15.222.65:33761), b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797)
I20260809 03:30:38.996311 28878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2914ee58a8624639b398c05117b14ecb" candidate_uuid: "91e7b9fd9bd64c459a270c1f30362a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3d706b01614f74af26febc495994ca"
I20260809 03:30:38.996362 29337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2914ee58a8624639b398c05117b14ecb" candidate_uuid: "91e7b9fd9bd64c459a270c1f30362a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23f891195a24a1aa460536b0a56da2f"
I20260809 03:30:38.996441 28878 raft_consensus.cc:3037] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.996464 29337 raft_consensus.cc:3037] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:38.997543 28878 raft_consensus.cc:2454] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e7b9fd9bd64c459a270c1f30362a89 in term 1.
I20260809 03:30:38.997543 29337 raft_consensus.cc:2454] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e7b9fd9bd64c459a270c1f30362a89 in term 1.
I20260809 03:30:38.997807 28992 leader_election.cc:302] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e7b9fd9bd64c459a270c1f30362a89, b23f891195a24a1aa460536b0a56da2f; no voters:
I20260809 03:30:38.997962 29485 raft_consensus.cc:2781] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:38.998123 29485 raft_consensus.cc:686] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 LEADER]: Becoming Leader. State: Replica: 91e7b9fd9bd64c459a270c1f30362a89, State: Running, Role: LEADER
I20260809 03:30:38.998358 29485 consensus_queue.cc:237] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } }
I20260809 03:30:39.000559 28628 catalog_manager.cc:5495] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "91e7b9fd9bd64c459a270c1f30362a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91e7b9fd9bd64c459a270c1f30362a89" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 } health_report { overall_health: UNKNOWN } } }
I20260809 03:30:39.024880 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.021695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.15.222.66:37705)
0809 03:30:39.021964 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:39.021970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:39.021982 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:39.022050 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:39.022052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:39.022100 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:39.022200 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:39.022366 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.022883 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.023642 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.023877 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.023947 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:39.024123 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:39.024596 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:39.024680 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:39.024708 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 03:30:39.024753 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
I20260809 03:30:39.033643 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.031149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46333 (local address 127.15.222.65:33761)
0809 03:30:39.031250 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:39.031254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:39.031298 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:30:39.031403 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:39.031405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:39.031446 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:39.031505 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:39.031675 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.032142 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.032880 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.033079 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.033138 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:39.033346 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:39.033405 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:39.033448 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:39.033506 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 03:30:39.033542 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:30:39.033664 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.030978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.15.222.66:46333)
0809 03:30:39.031234 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:39.031246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:39.031327 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:39.031514 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:39.031516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:39.031521 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:39.031608 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:39.031612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.032205 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.032207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:39.032802 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:39.032809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.033156 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.033159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:39.033218 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:39.033256 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:39.033456 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:39.033460 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:39.033499 (+ 39us) client_negotiation.cc:240] Negotiation successful
0809 03:30:39.033551 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260809 03:30:39.038910 28878 raft_consensus.cc:1261] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Refusing update from remote peer 91e7b9fd9bd64c459a270c1f30362a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:39.038956 29337 raft_consensus.cc:1261] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Refusing update from remote peer 91e7b9fd9bd64c459a270c1f30362a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:30:39.039358 29485 consensus_queue.cc:1035] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23f891195a24a1aa460536b0a56da2f" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:39.039856 29600 consensus_queue.cc:1035] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb3d706b01614f74af26febc495994ca" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:30:39.055186 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.052015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:38879 (local address 127.15.222.66:37705)
0809 03:30:39.052115 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:39.052120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:39.052312 (+ 192us) server_negotiation.cc:407] Connection header received
0809 03:30:39.052401 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:39.052404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:39.052453 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:39.052518 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:30:39.052768 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.053365 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.054208 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.054467 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.054555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:39.054781 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:30:39.054862 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:30:39.054919 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:39.055019 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:30:39.055070 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:30:39.055188 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.051953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:37705 (local address 127.15.222.65:38879)
0809 03:30:39.052228 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:39.052241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:39.052324 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:39.052549 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:39.052552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:39.052560 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 03:30:39.052666 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:39.052673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.053459 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.053462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:39.054108 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:39.054118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:39.054552 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:39.054555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:39.054654 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:39.054687 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 03:30:39.054938 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 03:30:39.054944 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:39.055007 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:30:39.055073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":67,"threads_started":1}
I20260809 03:30:39.077059 28622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:39.077205 28622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:30:39.079250 28622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99419ca5d354519b0cc1a622fae2b3b: Sending DeleteTablet for 3 replicas of tablet 2914ee58a8624639b398c05117b14ecb
I20260809 03:30:39.079775 29075 tablet_service.cc:1514] Processing DeleteTablet for tablet 2914ee58a8624639b398c05117b14ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:39 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260809 03:30:39.079830 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet 2914ee58a8624639b398c05117b14ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:39 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:60894
I20260809 03:30:39.079947 29507 tablet_replica.cc:330] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: stopping tablet replica
I20260809 03:30:39.080050 29507 raft_consensus.cc:2229] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:30:39.080090 29509 tablet_replica.cc:330] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: stopping tablet replica
I20260809 03:30:39.080192 29509 raft_consensus.cc:2229] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:39.080291 29509 raft_consensus.cc:2258] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:39.080348 29507 raft_consensus.cc:2258] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:39.080638 28850 tablet_service.cc:1514] Processing DeleteTablet for tablet 2914ee58a8624639b398c05117b14ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:30:39 UTC) from {username='oryx', principal='oryx/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:30:39.080813 29508 tablet_replica.cc:330] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca: stopping tablet replica
I20260809 03:30:39.080909 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 28739
I20260809 03:30:39.080919 29508 raft_consensus.cc:2229] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:30:39.081025 29508 raft_consensus.cc:2258] T 2914ee58a8624639b398c05117b14ecb P eb3d706b01614f74af26febc495994ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:30:39.091315 29507 ts_tablet_manager.cc:1905] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:39.091822 29509 ts_tablet_manager.cc:1905] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:30:39.093982 29507 ts_tablet_manager.cc:1918] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:39.094061 29507 log.cc:1196] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/2914ee58a8624639b398c05117b14ecb
I20260809 03:30:39.094293 29507 ts_tablet_manager.cc:1939] T 2914ee58a8624639b398c05117b14ecb P 91e7b9fd9bd64c459a270c1f30362a89: Deleting consensus metadata
I20260809 03:30:39.095232 28601 catalog_manager.cc:4841] TS 91e7b9fd9bd64c459a270c1f30362a89 (127.15.222.66:37705): tablet 2914ee58a8624639b398c05117b14ecb (table test-table [id=c07fa7f256ab49ad8377613556805ff5]) successfully deleted
I20260809 03:30:39.098778 29509 ts_tablet_manager.cc:1918] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 03:30:39.098866 29509 log.cc:1196] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/2914ee58a8624639b398c05117b14ecb
I20260809 03:30:39.099108 29509 ts_tablet_manager.cc:1939] T 2914ee58a8624639b398c05117b14ecb P b23f891195a24a1aa460536b0a56da2f: Deleting consensus metadata
I20260809 03:30:39.099826 28604 catalog_manager.cc:4841] TS b23f891195a24a1aa460536b0a56da2f (127.15.222.67:46797): tablet 2914ee58a8624639b398c05117b14ecb (table test-table [id=c07fa7f256ab49ad8377613556805ff5]) successfully deleted
W20260809 03:30:39.102659 28604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:33761 (error 108)
I20260809 03:30:39.102952 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 28970
I20260809 03:30:39.105636 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:39.103743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33761 (local address 127.0.0.1:52258)
0809 03:30:39.105460 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:39.105536 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:33761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":1480,"threads_started":1}
W20260809 03:30:39.105793 28604 catalog_manager.cc:4584] TS eb3d706b01614f74af26febc495994ca (127.15.222.65:33761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2914ee58a8624639b398c05117b14ecb: Network error: Client connection negotiation failed: client connection to 127.15.222.65:33761: connect: Connection refused (error 111)
I20260809 03:30:39.114316 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 29198
I20260809 03:30:39.126140 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 28576
2026-08-09T03:30:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3387 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:39 dist-test-slave-cvwc krb5kdc[29630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:39 dist-test-slave-cvwc krb5kdc[29630](info): set up 2 sockets
Aug 09 03:30:39 dist-test-slave-cvwc krb5kdc[29630](info): commencing operation
krb5kdc: starting...
W20260809 03:30:40.259802 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:40 dist-test-slave-cvwc krb5kdc[29630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:40.374135 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44947
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:40.483136 29650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:40.483385 29650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:40.483422 29650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:40.487345 29650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:40.487401 29650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:40.487417 29650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:40.487433 29650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:40.487447 29650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:40.492480 29650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:40.493389 29650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:40.494391 29650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:40.502256 29657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.502347 29658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:40.502272 29662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:40.502472 29650 server_base.cc:1034] running on GCE node
I20260809 03:30:40.503180 29650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:40.504170 29650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:40.505314 29650 hybrid_clock.cc:648] HybridClock initialized: now 1786246240505293 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T03:30:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1392 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:30:40 dist-test-slave-cvwc krb5kdc[29670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:30:40 dist-test-slave-cvwc krb5kdc[29670](info): set up 2 sockets
Aug 09 03:30:40 dist-test-slave-cvwc krb5kdc[29670](info): commencing operation
krb5kdc: starting...
W20260809 03:30:41.659142 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:30:41 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:30:41.774617 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35193
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:35193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:41.881930 29687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:41.882166 29687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:41.882202 29687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:41.886065 29687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:41.886121 29687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:41.886138 29687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:41.886154 29687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:41.886168 29687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:41.891338 29687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:35193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:41.892241 29687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:41.893167 29687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:41.901264 29687 server_base.cc:1034] running on GCE node
W20260809 03:30:41.901255 29693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:41.901228 29694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:41.901222 29696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:41.902122 29687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:41.903162 29687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:41.904312 29687 hybrid_clock.cc:648] HybridClock initialized: now 1786246241904294 us; error 31 us; skew 500 ppm
Aug 09 03:30:41 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:41.906961 29687 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:41.907828 29687 webserver.cc:460] Webserver started at http://127.15.222.126:39027/ using document root <none> and password file <none>
I20260809 03:30:41.908357 29687 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:41.908427 29687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:41.908632 29687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:41.910689 29687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "67ca44971c2d4b4cb630efd8a0f64542"
format_stamp: "Formatted at 2026-08-09 03:30:41 on dist-test-slave-cvwc"
server_key: "ad7d65cf72c1f2c0804762694af3c2cd"
server_key_iv: "4d2bf34e1a69b7701ff629248e250f72"
server_key_version: "encryptionkey@0"
I20260809 03:30:41.911244 29687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "67ca44971c2d4b4cb630efd8a0f64542"
format_stamp: "Formatted at 2026-08-09 03:30:41 on dist-test-slave-cvwc"
server_key: "ad7d65cf72c1f2c0804762694af3c2cd"
server_key_iv: "4d2bf34e1a69b7701ff629248e250f72"
server_key_version: "encryptionkey@0"
I20260809 03:30:41.914764 29687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:30:41.916992 29703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:41.918159 29687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:41.918268 29687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "67ca44971c2d4b4cb630efd8a0f64542"
format_stamp: "Formatted at 2026-08-09 03:30:41 on dist-test-slave-cvwc"
server_key: "ad7d65cf72c1f2c0804762694af3c2cd"
server_key_iv: "4d2bf34e1a69b7701ff629248e250f72"
server_key_version: "encryptionkey@0"
I20260809 03:30:41.918355 29687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:41.933231 29687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:41.936051 29687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:41.936245 29687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:41.943276 29687 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:35193
I20260809 03:30:41.943308 29792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:35193 every 8 connection(s)
I20260809 03:30:41.944175 29687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:41.946832 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:41.950253 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 29687
I20260809 03:30:41.950368 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:41.950685 16249 external_mini_cluster.cc:1428] Setting key 87574fe558ebd8eaaa6d484360d9e8e7
I20260809 03:30:41.952518 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Bootstrap starting.
I20260809 03:30:41.954840 29793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:41.955518 29793 log.cc:824] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:41.957199 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: No bootstrap required, opened a new log
Aug 09 03:30:41 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:41.961196 29793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:41.961462 29793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:41.961577 29793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ca44971c2d4b4cb630efd8a0f64542, State: Initialized, Role: FOLLOWER
I20260809 03:30:41.962069 29793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:41.962229 29793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:41.962293 29793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:41.962390 29793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:41.963235 29793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:41.963565 29793 leader_election.cc:302] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ca44971c2d4b4cb630efd8a0f64542; no voters:
I20260809 03:30:41.963864 29793 leader_election.cc:288] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:41.963944 29799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:41.964178 29799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 LEADER]: Becoming Leader. State: Replica: 67ca44971c2d4b4cb630efd8a0f64542, State: Running, Role: LEADER
I20260809 03:30:41.964570 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:41.952007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.15.222.126:35193)
0809 03:30:41.952405 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:41.952420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:41.952457 (+ 37us) server_negotiation.cc:407] Connection header received
0809 03:30:41.953078 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:41.953094 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:41.953416 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:41.953679 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:41.954545 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:41.955561 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:41.956415 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:41.956880 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:41.957022 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:41.959554 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:41.959561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:41.959579 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:41.959613 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:41.961426 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:41.961931 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:41.961937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:41.961942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:41.962075 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:41.962360 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:41.962363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:41.962365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:41.962818 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:41.963017 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:41.963303 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 03:30:41.963607 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260809 03:30:41.964542 29799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:41.965160 29793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:41.965735 29799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ca44971c2d4b4cb630efd8a0f64542. Latest consensus state: current_term: 1 leader_uuid: "67ca44971c2d4b4cb630efd8a0f64542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } } }
I20260809 03:30:41.965852 29799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:41.966493 29800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ca44971c2d4b4cb630efd8a0f64542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } } }
I20260809 03:30:41.966620 29809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:41.966646 29800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:41.969928 29809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:41.975574 29809 catalog_manager.cc:1347] Generated new cluster ID: 0c2f61e45c814ae98de7a3cd2a8ce550
I20260809 03:30:41.975629 29809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:41.993373 29809 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:41.994310 29809 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:41.999899 29809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Generated new TSK 0
I20260809 03:30:42.000540 29809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:30:42.032675 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:42.158896 29837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:42.159128 29837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:42.159183 29837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:42.163045 29837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:42.163097 29837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:42.163172 29837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:42.168299 29837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:42.169245 29837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:42.170250 29837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:42.179265 29846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.179287 29844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.179316 29848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.179587 29837 server_base.cc:1034] running on GCE node
I20260809 03:30:42.179982 29837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:42.180593 29837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:42.181758 29837 hybrid_clock.cc:648] HybridClock initialized: now 1786246242181738 us; error 28 us; skew 500 ppm
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:42.184378 29837 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:30:42.185166 29837 webserver.cc:460] Webserver started at http://127.15.222.65:41719/ using document root <none> and password file <none>
I20260809 03:30:42.185683 29837 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:42.185777 29837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:42.185981 29837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:42.188019 29837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "056bbbd46bb04c1dad84a861afd1e463"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "957a84fedf728e07e94ea7e312fe16d0"
server_key_iv: "324546bc325675a874479c6db6bb8623"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.188585 29837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "056bbbd46bb04c1dad84a861afd1e463"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "957a84fedf728e07e94ea7e312fe16d0"
server_key_iv: "324546bc325675a874479c6db6bb8623"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.192034 29837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:42.194243 29858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.195351 29837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:42.195462 29837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "056bbbd46bb04c1dad84a861afd1e463"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "957a84fedf728e07e94ea7e312fe16d0"
server_key_iv: "324546bc325675a874479c6db6bb8623"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.195547 29837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:42.212678 29837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:42.215655 29837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:42.215823 29837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:42.216361 29837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:42.217154 29837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:42.217199 29837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.217249 29837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:42.217278 29837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.225656 29837 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44113
I20260809 03:30:42.225700 30047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44113 every 8 connection(s)
I20260809 03:30:42.226511 29837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:42.229216 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 29837
I20260809 03:30:42.229341 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:42.229672 16249 external_mini_cluster.cc:1428] Setting key bf50aed4f558a42dc3648dc938d43cfa
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:30:42.239830 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.228089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:37405 (local address 127.15.222.126:35193)
0809 03:30:42.228237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:42.228242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:42.229022 (+ 780us) server_negotiation.cc:407] Connection header received
0809 03:30:42.230565 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:42.230568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:42.230614 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:42.230695 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:42.232047 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.232560 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.233823 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.234160 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.234267 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.236611 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:42.236614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:42.236616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:42.236640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:42.237965 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.238424 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.238427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.238428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.238518 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.238820 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.238824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.238825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.239002 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:42.239087 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:42.239541 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 03:30:42.239667 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:30:42.240540 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.228455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:37405)
0809 03:30:42.228869 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:42.228919 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:42.230358 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:42.230835 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:42.230842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:42.231291 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:42.231903 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.231924 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.232676 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.232679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.233688 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.233697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.234285 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.234288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.234436 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.234951 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:42.234969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:42.236463 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:42.238069 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.238074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.238089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.238325 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.238600 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.238603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.238605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.238737 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.239099 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:42.239106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:42.239325 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 03:30:42.239480 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 03:30:42.239775 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260809 03:30:42.241616 30048 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35193
I20260809 03:30:42.241852 30048 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:30:42.242332 30048 heartbeater.cc:511] Master 127.15.222.126:35193 requested a full tablet report, sending...
I20260809 03:30:42.243842 29734 ts_manager.cc:194] Registered new tserver with Master: 056bbbd46bb04c1dad84a861afd1e463 (127.15.222.65:44113)
I20260809 03:30:42.245352 29734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:37405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:30:42.256675 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:42.365121 30055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:42.365412 30055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:42.365461 30055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:42.369486 30055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:42.369542 30055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:42.369611 30055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:42.374682 30055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:42.375696 30055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:42.376681 30055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:42.385706 30063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.385707 30064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.385778 30069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.386039 30055 server_base.cc:1034] running on GCE node
I20260809 03:30:42.386391 30055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:42.386971 30055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:42.388114 30055 hybrid_clock.cc:648] HybridClock initialized: now 1786246242388099 us; error 26 us; skew 500 ppm
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:42.390702 30055 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:30:42.391556 30055 webserver.cc:460] Webserver started at http://127.15.222.66:35319/ using document root <none> and password file <none>
I20260809 03:30:42.392076 30055 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:42.392140 30055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:42.392323 30055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:42.394306 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "b477531cfa7348b9baf037446e730234"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "5f5ca6f9a0f02150469c4947be648695"
server_key_iv: "bd2e038e93e82127a98dd2bbbc69e1eb"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.394841 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "b477531cfa7348b9baf037446e730234"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "5f5ca6f9a0f02150469c4947be648695"
server_key_iv: "bd2e038e93e82127a98dd2bbbc69e1eb"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.398219 30055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:30:42.400488 30076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.401619 30055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:42.401752 30055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "b477531cfa7348b9baf037446e730234"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "5f5ca6f9a0f02150469c4947be648695"
server_key_iv: "bd2e038e93e82127a98dd2bbbc69e1eb"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.401842 30055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:42.414134 30055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:42.416517 30055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:42.416695 30055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:42.417191 30055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:42.417980 30055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:42.418028 30055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.418066 30055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:42.418085 30055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.426363 30055 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:38807
I20260809 03:30:42.426381 30263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:38807 every 8 connection(s)
I20260809 03:30:42.427158 30055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:42.432245 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 30055
I20260809 03:30:42.432363 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:42.432637 16249 external_mini_cluster.cc:1428] Setting key 75768cd38ada0b7a6cb6636d944eacbf
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:42.440194 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.428769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40413 (local address 127.15.222.126:35193)
0809 03:30:42.428935 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:42.428941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:42.429650 (+ 709us) server_negotiation.cc:407] Connection header received
0809 03:30:42.430934 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:42.430937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:42.430984 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:42.431051 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:42.432094 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.432843 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.434114 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.434339 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.434436 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.436748 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:42.436752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:42.436754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:42.436783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:42.438147 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.438707 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.438710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.438711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.438784 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.439163 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.439165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.439166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.439335 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:42.439417 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:42.439970 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 03:30:42.440066 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 03:30:42.441279 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.429092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:40413)
0809 03:30:42.429509 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:42.429549 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:42.430750 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:42.431187 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:42.431194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:42.431514 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:42.431967 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.431979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.432953 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.432956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.434005 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.434014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.434468 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.434472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.434625 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.435140 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:42.435155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:42.436593 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:42.438258 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.438264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.438277 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.438612 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.438875 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.438878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.438880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.439068 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.439430 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:42.439436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:42.439733 (+ 297us) client_negotiation.cc:769] Sending connection context
0809 03:30:42.439954 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 03:30:42.440278 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260809 03:30:42.442353 30264 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35193
I20260809 03:30:42.442595 30264 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:42.443163 30264 heartbeater.cc:511] Master 127.15.222.126:35193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:30:42.444183 29734 ts_manager.cc:194] Registered new tserver with Master: b477531cfa7348b9baf037446e730234 (127.15.222.66:38807)
I20260809 03:30:42.444768 29734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:40413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:30:42.458856 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:42.569921 30271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:42.570420 30271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:42.570530 30271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:42.575601 30271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:42.575677 30271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:42.575758 30271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:42.583734 30271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:42.585031 30271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:42.586469 30271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:42.601670 30279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.602131 30280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.602293 30271 server_base.cc:1034] running on GCE node
W20260809 03:30:42.610455 30284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.611028 30271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:42.611931 30271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:42.613134 30271 hybrid_clock.cc:648] HybridClock initialized: now 1786246242613107 us; error 33 us; skew 500 ppm
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:42.616619 30271 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:30:42.617542 30271 webserver.cc:460] Webserver started at http://127.15.222.67:36539/ using document root <none> and password file <none>
I20260809 03:30:42.618160 30271 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:42.618229 30271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:42.618444 30271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:42.620568 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "ea90eaf60ad64747a9ef596914fac8f1"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "d1390f1751b0f1d9c3cb1cae7a96292c"
server_key_iv: "4dc5c08f6fd4ef7d6350e5cf587eeeb4"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.621119 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "ea90eaf60ad64747a9ef596914fac8f1"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "d1390f1751b0f1d9c3cb1cae7a96292c"
server_key_iv: "4dc5c08f6fd4ef7d6350e5cf587eeeb4"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.624830 30271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:42.627157 30296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.628293 30271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:42.628391 30271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "ea90eaf60ad64747a9ef596914fac8f1"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "d1390f1751b0f1d9c3cb1cae7a96292c"
server_key_iv: "4dc5c08f6fd4ef7d6350e5cf587eeeb4"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.628489 30271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:42.641803 30271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:42.644711 30271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:42.644862 30271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:42.645390 30271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:42.646209 30271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:42.646252 30271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.646294 30271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:42.646323 30271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.654579 30271 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:43619
I20260809 03:30:42.654609 30483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:43619 every 8 connection(s)
I20260809 03:30:42.655421 30271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:30:42.665076 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 30271
I20260809 03:30:42.665189 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:42.665509 16249 external_mini_cluster.cc:1428] Setting key fb13253d7b9adbf3e9e1368450bc0306
I20260809 03:30:42.668177 29797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.657094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37609 (local address 127.15.222.126:35193)
0809 03:30:42.657231 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:42.657235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:42.658030 (+ 795us) server_negotiation.cc:407] Connection header received
0809 03:30:42.659307 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:42.659310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:42.659367 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:42.659450 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:42.660568 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.661085 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.662211 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.662433 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.662523 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.664708 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:42.664711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:42.664713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:42.664738 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:42.666168 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.666761 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.666765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.666766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.666860 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:42.667202 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:42.667204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:42.667205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:42.667371 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:42.667510 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:42.667963 (+ 453us) server_negotiation.cc:299] Negotiation successful
0809 03:30:42.668060 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:30:42.668891 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:42.657421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:37609)
0809 03:30:42.657880 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:42.657935 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:42.659102 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:42.659627 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:42.659634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:42.659975 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:42.660422 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.660436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.661203 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.661206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.662091 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:42.662100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:42.662571 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:42.662574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:42.662740 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:42.663190 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:42.663205 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:42.664547 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:42.666276 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.666282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.666295 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.666671 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.666948 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:30:42.666951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:30:42.666952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:30:42.667124 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:30:42.667518 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:42.667524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:42.667765 (+ 241us) client_negotiation.cc:769] Sending connection context
0809 03:30:42.667918 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 03:30:42.668150 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260809 03:30:42.669883 30484 heartbeater.cc:347] Connected to a master server at 127.15.222.126:35193
I20260809 03:30:42.670095 30484 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:42.670529 30484 heartbeater.cc:511] Master 127.15.222.126:35193 requested a full tablet report, sending...
I20260809 03:30:42.671415 29734 ts_manager.cc:194] Registered new tserver with Master: ea90eaf60ad64747a9ef596914fac8f1 (127.15.222.67:43619)
I20260809 03:30:42.671943 29734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:37609
I20260809 03:30:42.678994 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
Entry for principal oryx/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab.
WARNING: no policy specified for oryx/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
Entry for principal oryx/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab.
WARNING: no policy specified for oryx/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
Entry for principal oryx/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab.
I20260809 03:30:42.744230 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 29837
I20260809 03:30:42.760201 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 30055
I20260809 03:30:42.778373 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 30271
I20260809 03:30:42.788442 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 29687
I20260809 03:30:42.796669 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35193
--webserver_interface=127.15.222.126
--webserver_port=39027
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:35193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:30:42.905308 30499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:42.905555 30499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:42.905594 30499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:42.909482 30499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:42.909531 30499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:42.909560 30499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:42.909591 30499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:42.909618 30499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:42.914721 30499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:35193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:42.915733 30499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:42.916769 30499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:42.924880 30505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:42.924890 30509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.925083 30499 server_base.cc:1034] running on GCE node
W20260809 03:30:42.925105 30506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:42.925697 30499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:42.926801 30499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:42.927959 30499 hybrid_clock.cc:648] HybridClock initialized: now 1786246242927944 us; error 25 us; skew 500 ppm
Aug 09 03:30:42 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:42.930513 30499 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:30:42.931380 30499 webserver.cc:460] Webserver started at http://127.15.222.126:39027/ using document root <none> and password file <none>
I20260809 03:30:42.931923 30499 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:42.932008 30499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:42.935702 30499 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 03:30:42.937634 30520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:42.938781 30499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:30:42.938901 30499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "67ca44971c2d4b4cb630efd8a0f64542"
format_stamp: "Formatted at 2026-08-09 03:30:41 on dist-test-slave-cvwc"
server_key: "ad7d65cf72c1f2c0804762694af3c2cd"
server_key_iv: "4d2bf34e1a69b7701ff629248e250f72"
server_key_version: "encryptionkey@0"
I20260809 03:30:42.939397 30499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:42.961297 30499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:42.964056 30499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:42.964233 30499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:42.971006 30499 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:35193
I20260809 03:30:42.971014 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:35193 every 8 connection(s)
I20260809 03:30:42.972045 30499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:42.972229 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 30499
I20260809 03:30:42.972673 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:44113
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:35193
--webserver_port=41719
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:30:42.976610 30598 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:30:42.978876 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Bootstrap starting.
I20260809 03:30:43.000939 30598 log.cc:824] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:43.009167 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:30:43.009599 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Bootstrap complete.
I20260809 03:30:43.015096 30598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:43.015625 30598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ca44971c2d4b4cb630efd8a0f64542, State: Initialized, Role: FOLLOWER
I20260809 03:30:43.016214 30598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:43.016384 30598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:43.016467 30598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:43.016571 30598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:30:43.019062 30598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:43.019431 30598 leader_election.cc:302] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ca44971c2d4b4cb630efd8a0f64542; no voters:
I20260809 03:30:43.019761 30598 leader_election.cc:288] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:30:43.019876 30606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:30:43.020120 30606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [term 2 LEADER]: Becoming Leader. State: Replica: 67ca44971c2d4b4cb630efd8a0f64542, State: Running, Role: LEADER
I20260809 03:30:43.020454 30606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } }
I20260809 03:30:43.021009 30598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:43.021839 30608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ca44971c2d4b4cb630efd8a0f64542. Latest consensus state: current_term: 2 leader_uuid: "67ca44971c2d4b4cb630efd8a0f64542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } } }
I20260809 03:30:43.021839 30607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67ca44971c2d4b4cb630efd8a0f64542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ca44971c2d4b4cb630efd8a0f64542" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35193 } } }
I20260809 03:30:43.021984 30608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:43.021991 30607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542 [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:43.022439 30617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:43.025557 30617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:43.026239 30617 catalog_manager.cc:1259] Loaded cluster ID: 0c2f61e45c814ae98de7a3cd2a8ce550
I20260809 03:30:43.026290 30617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:43.027937 30617 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:43.028483 30617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67ca44971c2d4b4cb630efd8a0f64542: Loaded TSK: 0
I20260809 03:30:43.029363 30617 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:30:43.122563 30602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:43.122790 30602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:43.122855 30602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:43.126813 30602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:43.126866 30602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:43.126940 30602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:43.132030 30602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.65.keytab
--principal=oryx/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:44113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=41719
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:43.132982 30602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:43.133980 30602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:43.142897 30644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.142930 30639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.142947 30640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:43.143234 30602 server_base.cc:1034] running on GCE node
I20260809 03:30:43.143714 30602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:43.144479 30602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:43.145671 30602 hybrid_clock.cc:648] HybridClock initialized: now 1786246243145649 us; error 35 us; skew 500 ppm
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246243, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:43.148895 30602 init.cc:377] Logged in from keytab as oryx/127.15.222.65@KRBTEST.COM (short username oryx)
I20260809 03:30:43.149981 30602 webserver.cc:460] Webserver started at http://127.15.222.65:41719/ using document root <none> and password file <none>
I20260809 03:30:43.150637 30602 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:43.150736 30602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:43.155637 30602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 03:30:43.158062 30655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.159468 30602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:43.159585 30602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "056bbbd46bb04c1dad84a861afd1e463"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "957a84fedf728e07e94ea7e312fe16d0"
server_key_iv: "324546bc325675a874479c6db6bb8623"
server_key_version: "encryptionkey@0"
I20260809 03:30:43.160276 30602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:43.177193 30602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:43.180480 30602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:43.180678 30602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:43.181347 30602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:43.182377 30602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:43.182435 30602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.182483 30602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:43.182511 30602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.192960 30602 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44113
I20260809 03:30:43.192974 30827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44113 every 8 connection(s)
I20260809 03:30:43.194180 30602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:43.200249 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 30602
I20260809 03:30:43.200672 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:38807
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:35193
--webserver_port=35319
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:30:43.207830 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.196275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:57539 (local address 127.15.222.126:35193)
0809 03:30:43.196687 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.196704 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.197112 (+ 408us) server_negotiation.cc:407] Connection header received
0809 03:30:43.198635 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.198646 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.198942 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.199238 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.200465 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.201550 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.202940 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.203657 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.203815 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.207142 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:57539: BlockingRecv error: recv got EOF from 127.15.222.65:57539 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":91,"threads_started":1}
I20260809 03:30:43.207840 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.196508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:57539)
0809 03:30:43.196955 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.196996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.198223 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.199340 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.199348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.199745 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.200290 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.200311 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.201684 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.201687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.202803 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.202814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.203810 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.203813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.203947 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.204491 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.204507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.206913 (+ 2406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
W20260809 03:30:43.208346 30828 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:35193 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.222.126:35193: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.213867 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.208747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:53071)
0809 03:30:43.208895 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.208911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.208992 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.209208 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.209211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.209369 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.209480 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.209487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.210203 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.210206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.211033 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.211044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.211431 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.211434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.211525 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.211939 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.211954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.213743 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 03:30:43.213973 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.208860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:53071 (local address 127.15.222.126:35193)
0809 03:30:43.208960 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.208963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.208974 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:30:43.209074 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.209076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.209112 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.209171 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.209579 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.210106 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.211135 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.211336 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.211402 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.213865 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:53071: BlockingRecv error: recv got EOF from 127.15.222.65:53071 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.219422 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.214530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:44115)
0809 03:30:43.214669 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.214681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.214743 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.214937 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.214940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.215086 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.215168 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.215173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.216035 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.216038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.216863 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.216873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.217267 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.217270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.217347 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.217738 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.217752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.219314 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260809 03:30:43.219548 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.214601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:44115 (local address 127.15.222.126:35193)
0809 03:30:43.214693 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.214696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.214711 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:43.214811 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.214813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.214847 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.214910 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.215250 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.215836 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.216985 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.217184 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.217254 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.219455 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:44115: BlockingRecv error: recv got EOF from 127.15.222.65:44115 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 03:30:43.219686 30828 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:30:43.315065 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:43.315286 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:43.315323 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:43.319121 30839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:43.319164 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:43.319216 30839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:43.324229 30839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.66.keytab
--principal=oryx/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:38807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=35319
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:43.325096 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:43.325999 30839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:30:43.334846 30839 server_base.cc:1034] running on GCE node
W20260809 03:30:43.334800 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.334797 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.334817 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:43.335395 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:43.335961 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:43.337113 30839 hybrid_clock.cc:648] HybridClock initialized: now 1786246243337091 us; error 32 us; skew 500 ppm
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246243, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:43.339641 30839 init.cc:377] Logged in from keytab as oryx/127.15.222.66@KRBTEST.COM (short username oryx)
I20260809 03:30:43.340390 30839 webserver.cc:460] Webserver started at http://127.15.222.66:35319/ using document root <none> and password file <none>
I20260809 03:30:43.340876 30839 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:43.340955 30839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:43.344696 30839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:43.346570 30857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.347697 30839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:43.347793 30839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "b477531cfa7348b9baf037446e730234"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "5f5ca6f9a0f02150469c4947be648695"
server_key_iv: "bd2e038e93e82127a98dd2bbbc69e1eb"
server_key_version: "encryptionkey@0"
I20260809 03:30:43.348279 30839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:43.374711 30839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:43.377447 30839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:43.377619 30839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:43.378283 30839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:43.379159 30839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:43.379210 30839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.379256 30839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:43.379279 30839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.388288 30839 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:38807
I20260809 03:30:43.388317 31033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:38807 every 8 connection(s)
I20260809 03:30:43.389314 30839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:43.398365 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 30839
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.398792 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:43619
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:35193
--webserver_port=36539
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:30:43.400381 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.391331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34393 (local address 127.15.222.126:35193)
0809 03:30:43.391482 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.391488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.392391 (+ 903us) server_negotiation.cc:407] Connection header received
0809 03:30:43.393680 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.393683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.393770 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.393863 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.395036 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.395606 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.396667 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.396902 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.396997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.400255 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:34393: BlockingRecv error: recv got EOF from 127.15.222.66:34393 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:30:43.401027 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.391780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:34393)
0809 03:30:43.392234 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.392275 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.393463 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.394038 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.394045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.394394 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.394888 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.394903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.395708 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.395711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.396555 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.396564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.397022 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.397024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.397159 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.397685 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.397702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.400119 (+ 2417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
W20260809 03:30:43.401525 31034 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:35193 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.222.126:35193: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.406548 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.401944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:57093)
0809 03:30:43.402080 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.402093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.402163 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.402380 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.402383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.402533 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.402617 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.402622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.403302 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.403305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.403831 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.403838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.404245 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.404257 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.404341 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.404804 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.404814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.406432 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260809 03:30:43.406668 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.402045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:57093 (local address 127.15.222.126:35193)
0809 03:30:43.402158 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.402162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.402174 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:43.402248 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.402251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.402292 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.402355 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.402721 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.403203 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.403933 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.404153 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.404235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.406564 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:57093: BlockingRecv error: recv got EOF from 127.15.222.66:57093 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.411550 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.407188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:46035)
0809 03:30:43.407337 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.407350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.407420 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.407628 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.407631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.407765 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.407835 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.407840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.408508 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.408511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.409000 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.409006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.409403 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.409406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.409477 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.409917 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.409927 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.411434 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260809 03:30:43.411621 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.407287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46035 (local address 127.15.222.126:35193)
0809 03:30:43.407400 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.407403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.407417 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:30:43.407506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.407509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.407548 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.407604 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.407937 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.408417 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.409098 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.409314 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.409396 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.411536 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:46035: BlockingRecv error: recv got EOF from 127.15.222.66:46035 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 03:30:43.411891 31034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:30:43.509794 31038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:43.510038 31038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:43.510083 31038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:43.514186 31038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:43.514240 31038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:43.514334 31038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:43.519388 31038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/krb5kdc/oryx_127.15.222.67.keytab
--principal=oryx/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:43619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=36539
--tserver_master_addrs=127.15.222.126:35193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:43.520323 31038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:43.521324 31038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:43.530711 31047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.530695 31045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:43.530696 31050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:43.530860 31038 server_base.cc:1034] running on GCE node
I20260809 03:30:43.531329 31038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:43.531929 31038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:43.533109 31038 hybrid_clock.cc:648] HybridClock initialized: now 1786246243533083 us; error 39 us; skew 500 ppm
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246243, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:30:43.535697 31038 init.cc:377] Logged in from keytab as oryx/127.15.222.67@KRBTEST.COM (short username oryx)
I20260809 03:30:43.536515 31038 webserver.cc:460] Webserver started at http://127.15.222.67:36539/ using document root <none> and password file <none>
I20260809 03:30:43.537017 31038 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:43.537096 31038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:43.540656 31038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:43.542575 31063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.543763 31038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:43.543851 31038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "ea90eaf60ad64747a9ef596914fac8f1"
format_stamp: "Formatted at 2026-08-09 03:30:42 on dist-test-slave-cvwc"
server_key: "d1390f1751b0f1d9c3cb1cae7a96292c"
server_key_iv: "4dc5c08f6fd4ef7d6350e5cf587eeeb4"
server_key_version: "encryptionkey@0"
I20260809 03:30:43.544327 31038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:43.560448 31038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:43.562928 31038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:43.563066 31038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:43.563565 31038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:43.564296 31038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:43.564340 31038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.564378 31038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:43.564404 31038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:43.572505 31038 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:43619
I20260809 03:30:43.572530 31226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:43619 every 8 connection(s)
I20260809 03:30:43.573387 31038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:43.575819 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 31038
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 03:30:43.583902 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.575025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:45675 (local address 127.15.222.126:35193)
0809 03:30:43.575160 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.575165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.575979 (+ 814us) server_negotiation.cc:407] Connection header received
0809 03:30:43.577493 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.577496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.577539 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.577608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.578999 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.579466 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.580530 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.580731 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.580812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.583772 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:45675: BlockingRecv error: recv got EOF from 127.15.222.67:45675 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:43.584532 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.575400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:45675)
0809 03:30:43.575838 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.575880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.577293 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.577808 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.577815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.578251 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.578873 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.578890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.579577 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.579580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.580416 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.580425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.580860 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.580864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.581030 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.581517 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.581530 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.583659 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260809 03:30:43.585146 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.576751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.15.222.126:35193)
0809 03:30:43.577001 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.577005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.577020 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:43.577065 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.577068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.577114 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.577207 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.577903 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.578690 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.579504 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.579732 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.579819 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.581110 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:43.581116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:43.581129 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:30:43.581156 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:43.582789 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:43.583220 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:43.583227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:43.583233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:43.583348 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:30:43.583602 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:30:43.583605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:30:43.583607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:30:43.583929 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:43.584225 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:43.584589 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 03:30:43.584992 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
W20260809 03:30:43.585182 31227 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:35193 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.222.126:35193: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.590009 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.585560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:50113)
0809 03:30:43.585659 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.585673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.585760 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.585970 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.585972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.586103 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.586203 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.586209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.586911 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.586915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.587580 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.587591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.588022 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.588025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.588091 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.588405 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.588415 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.589894 (+ 1479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":22}
W20260809 03:30:43.590111 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.585613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:50113 (local address 127.15.222.126:35193)
0809 03:30:43.585706 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.585731 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.585744 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:43.585830 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.585832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.585870 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.585940 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.586317 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.586795 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.587687 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.587922 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.587993 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.590017 (+ 2024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:50113: BlockingRecv error: recv got EOF from 127.15.222.67:50113 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:43 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:43.594213 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:43.590484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:50591)
0809 03:30:43.590566 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:43.590577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:43.590633 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:43.590807 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:43.590809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:43.590918 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:43.590977 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.590982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.591563 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.591566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.592037 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:43.592043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.592358 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.592360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:43.592416 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.592719 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:43.592729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:43.594137 (+ 1408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260809 03:30:43.594339 30831 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:43.590550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:50591 (local address 127.15.222.126:35193)
0809 03:30:43.590662 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:43.590666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:43.590678 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:43.590704 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:43.590706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:43.590734 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:43.590790 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:43.591051 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.591492 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.592110 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:43.592292 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:43.592352 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:43.594239 (+ 1887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:50591: BlockingRecv error: recv got EOF from 127.15.222.67:50591 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 03:30:43.594365 31227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:44.225260 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:44.220224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:59823)
0809 03:30:44.220473 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:44.220489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:44.220581 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:44.220789 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:44.220791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:44.220966 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:44.221091 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.221097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.221865 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.221868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.222480 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.222490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.222905 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.222909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.222982 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.223483 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:44.223497 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:44.225151 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
W20260809 03:30:44.225409 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:44.220291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:59823 (local address 127.15.222.126:35193)
0809 03:30:44.220513 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:44.220517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:44.220533 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:30:44.220661 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:44.220663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:44.220704 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:44.220766 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:44.221173 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.221776 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.222575 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.222810 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.222878 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.225281 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:59823: BlockingRecv error: recv got EOF from 127.15.222.65:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:44.417629 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:44.412453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:48123)
0809 03:30:44.412695 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:44.412712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:44.412804 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:44.413058 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:44.413061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:44.413294 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:44.413452 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.413463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.414356 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.414359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.414924 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.414933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.415287 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.415289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.415353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.415787 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:44.415797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:44.417519 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
W20260809 03:30:44.417768 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:44.412513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:48123 (local address 127.15.222.126:35193)
0809 03:30:44.412629 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:44.412633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:44.412770 (+ 137us) server_negotiation.cc:407] Connection header received
0809 03:30:44.412870 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:44.412872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:44.412922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:44.413010 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:44.413584 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.414264 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.415011 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.415213 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.415279 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.417644 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:48123: BlockingRecv error: recv got EOF from 127.15.222.66:48123 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:44 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:44.599907 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:44.594924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:60157)
0809 03:30:44.595170 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:44.595183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:44.595280 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:44.595480 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:44.595482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:44.595658 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:44.595776 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.595781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.596443 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.596445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.597000 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:44.597007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.597357 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.597359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:44.597440 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.597965 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:44.597976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:44.599779 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260809 03:30:44.600070 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:44.594971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:60157 (local address 127.15.222.126:35193)
0809 03:30:44.595088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:44.595092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:44.595241 (+ 149us) server_negotiation.cc:407] Connection header received
0809 03:30:44.595341 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:44.595343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:44.595388 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:44.595452 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:44.595856 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.596370 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.597080 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:44.597282 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:44.597348 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:44.599944 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:60157: BlockingRecv error: recv got EOF from 127.15.222.67:60157 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:45.230984 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:45.225990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:50079)
0809 03:30:45.226258 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:45.226271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:45.226358 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:45.226619 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:45.226621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:45.226801 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:45.226911 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.226917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.227625 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.227627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.228175 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.228181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.228564 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.228567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.228622 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.229099 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:45.229111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:45.230870 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
W20260809 03:30:45.231159 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:45.226047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:50079 (local address 127.15.222.126:35193)
0809 03:30:45.226312 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:45.226316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:45.226331 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:45.226457 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:45.226460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:45.226508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:45.226590 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:45.227007 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.227547 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.228264 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.228482 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.228560 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.231019 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:50079: BlockingRecv error: recv got EOF from 127.15.222.65:50079 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:45.423195 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:45.418325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:56779)
0809 03:30:45.418583 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:45.418596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:45.418691 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:45.418893 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:45.418895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:45.419073 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:45.419168 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.419173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.419831 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.419833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.420409 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.420416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.420798 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.420800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.420897 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.421335 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:45.421346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:45.423080 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260809 03:30:45.423321 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:45.418392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:56779 (local address 127.15.222.126:35193)
0809 03:30:45.418538 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:45.418543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:45.418655 (+ 112us) server_negotiation.cc:407] Connection header received
0809 03:30:45.418758 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:45.418760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:45.418804 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:45.418867 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:45.419250 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.419756 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.420496 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.420717 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.420792 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.423204 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:56779: BlockingRecv error: recv got EOF from 127.15.222.66:56779 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:45 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:45.608448 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:45.600624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:50757)
0809 03:30:45.600871 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:45.600884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:45.600978 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:45.601175 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:45.601177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:45.601351 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:45.601466 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.601471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.602148 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.602150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.602698 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:45.602705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.603053 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.603055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:45.603108 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.606602 (+ 3494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:45.606615 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:45.608323 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
W20260809 03:30:45.608664 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:45.600752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:50757 (local address 127.15.222.126:35193)
0809 03:30:45.600869 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:45.600873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:45.600933 (+ 60us) server_negotiation.cc:407] Connection header received
0809 03:30:45.601040 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:45.601042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:45.601086 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:45.601150 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:45.601550 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.602079 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.602779 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:45.602979 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:45.603043 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:45.608476 (+ 5433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:50757: BlockingRecv error: recv got EOF from 127.15.222.67:50757 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:46.236939 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:46.231665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:59355)
0809 03:30:46.231941 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:46.231954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:46.232051 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:46.232258 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:46.232260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:46.232436 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:46.232536 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.232541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.233206 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.233208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.233958 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.233970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.234505 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.234509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.234584 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.235034 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:46.235044 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:46.236771 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
W20260809 03:30:46.237012 31254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:46.231750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:59355 (local address 127.15.222.126:35193)
0809 03:30:46.231965 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:46.231968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:46.232016 (+ 48us) server_negotiation.cc:407] Connection header received
0809 03:30:46.232118 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:46.232120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:46.232162 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:46.232235 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:46.232615 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.233131 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.234074 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.234374 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.234489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.236901 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:59355: BlockingRecv error: recv got EOF from 127.15.222.65:59355 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:46.428797 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:46.423859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:53511)
0809 03:30:46.424114 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:46.424128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:46.424234 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:46.424439 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:46.424441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:46.424617 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:46.424714 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.424719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.425465 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.425467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.426077 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.426087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.426449 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.426451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.426506 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.426942 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:46.426953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:46.428672 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260809 03:30:46.428887 31254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:46.423918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:53511 (local address 127.15.222.126:35193)
0809 03:30:46.424049 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:46.424053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:46.424196 (+ 143us) server_negotiation.cc:407] Connection header received
0809 03:30:46.424303 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:46.424305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:46.424350 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:46.424416 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:46.424838 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.425376 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.426164 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.426373 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.426441 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.428791 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:53511: BlockingRecv error: recv got EOF from 127.15.222.66:53511 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:46 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:46.614657 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:46.609155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:37181)
0809 03:30:46.609438 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:46.609454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:46.609566 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:46.609819 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:46.609822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:46.610031 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:46.610127 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.610132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.610893 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.610897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.611462 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:46.611471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.611918 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.611922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:46.612006 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.612503 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:46.612514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:46.614530 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
W20260809 03:30:46.614746 31254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:46.609251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:37181 (local address 127.15.222.126:35193)
0809 03:30:46.609406 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:46.609411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:46.609522 (+ 111us) server_negotiation.cc:407] Connection header received
0809 03:30:46.609649 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:46.609652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:46.609701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:46.609785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:46.610237 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.610779 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.611575 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:46.611807 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:46.611895 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:46.614652 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:37181: BlockingRecv error: recv got EOF from 127.15.222.67:37181 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:47.242811 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:47.237642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:44169)
0809 03:30:47.237942 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:47.237956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:47.238053 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:47.238258 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:47.238260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:47.238433 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:47.238542 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.238547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.239267 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.239270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.240019 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.240031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.240456 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.240459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.240522 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.240947 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:47.240957 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:47.242698 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
W20260809 03:30:47.242996 31258 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:47.237705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:44169 (local address 127.15.222.126:35193)
0809 03:30:47.237969 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:47.237972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:47.238006 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:30:47.238121 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:47.238123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:47.238164 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:47.238234 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:47.238625 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.239163 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.240129 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.240353 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.240441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.242822 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:44169: BlockingRecv error: recv got EOF from 127.15.222.65:44169 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:47.434340 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:47.429484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:54335)
0809 03:30:47.429759 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:47.429774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:47.429865 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:47.430119 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:47.430122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:47.430300 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:47.430415 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.430422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.431115 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.431118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.431675 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.431684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.432050 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.432053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.432126 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.432563 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:47.432574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:47.434224 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
W20260809 03:30:47.434453 31258 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:47.429555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:54335 (local address 127.15.222.126:35193)
0809 03:30:47.429821 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:47.429825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:47.429838 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:47.429959 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:47.429962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:47.430016 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:47.430085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:47.430501 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.431011 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.431758 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.431959 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.432020 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.434346 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:54335: BlockingRecv error: recv got EOF from 127.15.222.66:54335 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:47 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:47.620471 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:47.615369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:43717)
0809 03:30:47.615682 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:47.615694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:47.615792 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:47.616005 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:47.616007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:47.616181 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:47.616278 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.616283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.616952 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.616954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.617504 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:47.617511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.617888 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.617890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:47.617947 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.618368 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:47.618378 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:47.620332 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":75,"threads_started":1}
W20260809 03:30:47.620581 31258 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:47.615427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:43717 (local address 127.15.222.126:35193)
0809 03:30:47.615553 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:47.615557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:47.615754 (+ 197us) server_negotiation.cc:407] Connection header received
0809 03:30:47.615859 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:47.615861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:47.615914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:47.615979 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:47.616357 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.616864 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.617579 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:47.617815 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:47.617887 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:47.620471 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:43717: BlockingRecv error: recv got EOF from 127.15.222.67:43717 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:48.248692 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:48.243553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:59699)
0809 03:30:48.243806 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:48.243819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:48.243921 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:48.244129 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:48.244131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:48.244311 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:48.244412 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.244417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.245099 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.245101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.245670 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.245677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.246091 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.246094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.246156 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.246645 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:48.246657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:48.248576 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
W20260809 03:30:48.248816 31262 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:48.243605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:59699 (local address 127.15.222.126:35193)
0809 03:30:48.243818 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:48.243821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:48.243870 (+ 49us) server_negotiation.cc:407] Connection header received
0809 03:30:48.243990 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:48.243992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:48.244034 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:48.244104 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:48.244495 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.245020 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.245790 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.246011 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.246077 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.248704 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:59699: BlockingRecv error: recv got EOF from 127.15.222.65:59699 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:48.440229 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:48.435121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:57499)
0809 03:30:48.435379 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:48.435405 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:48.435497 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:48.435700 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:48.435702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:48.435899 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:48.435994 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.436000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.436793 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.436796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.437366 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.437375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.437835 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.437838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.437924 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.438354 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:48.438365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:48.440112 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
W20260809 03:30:48.440351 31262 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:48.435264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:57499 (local address 127.15.222.126:35193)
0809 03:30:48.435422 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:48.435425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:48.435453 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:30:48.435568 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:48.435570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:48.435610 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:48.435676 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:48.436106 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.436684 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.437490 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.437732 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.437827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.440238 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:57499: BlockingRecv error: recv got EOF from 127.15.222.66:57499 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:48 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:48.626268 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:48.621285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:58971)
0809 03:30:48.621566 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:48.621580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:48.621702 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:48.621953 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:48.621955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:48.622141 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:48.622238 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.622244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.622958 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.622961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.623535 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:48.623544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.623893 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.623896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:48.623952 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.624415 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:48.624426 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:48.626150 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
W20260809 03:30:48.626415 31262 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:48.621366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:58971 (local address 127.15.222.126:35193)
0809 03:30:48.621561 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:48.621566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:48.621657 (+ 91us) server_negotiation.cc:407] Connection header received
0809 03:30:48.621817 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:48.621819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:48.621867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:48.621923 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:48.622323 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.622862 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.623616 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:48.623817 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:48.623884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:48.626281 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:58971: BlockingRecv error: recv got EOF from 127.15.222.67:58971 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:49.254698 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:49.249420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:41703)
0809 03:30:49.249673 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:49.249686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:49.249791 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:49.250013 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:49.250015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:49.250195 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:49.250294 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.250299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.251068 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.251070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.251670 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.251680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.252111 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.252114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.252170 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.252632 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:49.252644 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:49.254565 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
W20260809 03:30:49.254849 31267 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:49.249479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:41703 (local address 127.15.222.126:35193)
0809 03:30:49.249805 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:49.249808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:49.249824 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:30:49.249863 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:49.249865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:49.249911 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:49.249998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:49.250416 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.250987 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.251784 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.252028 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.252105 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.254717 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:41703: BlockingRecv error: recv got EOF from 127.15.222.65:41703 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:49.446256 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:49.440943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:52011)
0809 03:30:49.441191 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:49.441206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:49.441308 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:49.441529 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:49.441532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:49.441748 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:49.441857 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.441863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.442620 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.442622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.443178 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.443189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.443663 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.443667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.443746 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.444188 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:49.444199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:49.446113 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
W20260809 03:30:49.446369 31267 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:49.440999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:52011 (local address 127.15.222.126:35193)
0809 03:30:49.441148 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:49.441152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:49.441272 (+ 120us) server_negotiation.cc:407] Connection header received
0809 03:30:49.441377 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:49.441379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:49.441434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:49.441501 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:49.441953 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.442530 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.443289 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.443553 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.443643 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.446255 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:52011: BlockingRecv error: recv got EOF from 127.15.222.66:52011 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:49 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:49.632174 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:49.626983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:56727)
0809 03:30:49.627278 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:49.627291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:49.627399 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:49.627640 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:49.627642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:49.627847 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:49.627951 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.627957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.628688 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.628692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.629246 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:49.629254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.629686 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.629689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:49.629810 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.630266 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:49.630278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:49.632030 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
W20260809 03:30:49.632278 31267 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:49.627063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:56727 (local address 127.15.222.126:35193)
0809 03:30:49.627194 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:49.627198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:49.627358 (+ 160us) server_negotiation.cc:407] Connection header received
0809 03:30:49.627486 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:49.627489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:49.627539 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:49.627607 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:49.628061 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.628588 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.629356 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:49.629584 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:49.629669 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:49.632165 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:56727: BlockingRecv error: recv got EOF from 127.15.222.67:56727 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:50.260246 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:50.255368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:47151)
0809 03:30:50.255663 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:50.255676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:50.255775 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:50.255987 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:50.255989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:50.256168 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:50.256267 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.256272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.256964 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.256967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.257508 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.257514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.257905 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.257907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.257971 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.258448 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:50.258460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:50.260130 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":69,"threads_started":1}
W20260809 03:30:50.260392 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:50.255434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47151 (local address 127.15.222.126:35193)
0809 03:30:50.255653 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:50.255656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:50.255735 (+ 79us) server_negotiation.cc:407] Connection header received
0809 03:30:50.255841 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:50.255843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:50.255888 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:50.255961 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:50.256346 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.256879 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.257585 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.257817 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.257884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.260252 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:47151: BlockingRecv error: recv got EOF from 127.15.222.65:47151 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:50.452046 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:50.447064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:60527)
0809 03:30:50.447348 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:50.447361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:50.447460 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:50.447672 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:50.447675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:50.447876 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:50.448008 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.448017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.448714 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.448716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.449293 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.449302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.449690 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.449692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.449792 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.450221 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:50.450231 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:50.451919 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
W20260809 03:30:50.452138 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:50.447108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:60527 (local address 127.15.222.126:35193)
0809 03:30:50.447244 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:50.447247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:50.447423 (+ 176us) server_negotiation.cc:407] Connection header received
0809 03:30:50.447529 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:50.447531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:50.447574 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:50.447639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:50.448104 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.448623 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.449382 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.449607 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.449675 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.452032 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:60527: BlockingRecv error: recv got EOF from 127.15.222.66:60527 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:50 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:50.642453 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:50.632938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:53293)
0809 03:30:50.633228 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:50.633241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:50.633349 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:50.633587 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:50.633589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:50.633811 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:50.633912 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.633918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.634682 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.634686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.635310 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:50.635320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.635798 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.635801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:50.635881 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.640535 (+ 4654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:50.640547 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:50.642304 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":68,"threads_started":1}
W20260809 03:30:50.642567 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:50.632999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:53293 (local address 127.15.222.126:35193)
0809 03:30:50.633136 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:50.633140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:50.633310 (+ 170us) server_negotiation.cc:407] Connection header received
0809 03:30:50.633436 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:50.633440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:50.633489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:50.633552 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:50.634026 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.634581 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.635425 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:50.635693 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:50.635782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:50.642440 (+ 6658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:53293: BlockingRecv error: recv got EOF from 127.15.222.67:53293 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:51.265820 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:51.260908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:46081)
0809 03:30:51.261187 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:51.261203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:51.261298 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:51.261498 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:51.261500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:51.261679 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:51.261801 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.261806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.262499 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.262501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.263044 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.263050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.263408 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.263410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.263465 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.263900 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:51.263910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:51.265680 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
W20260809 03:30:51.266026 31279 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:51.261061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:46081 (local address 127.15.222.126:35193)
0809 03:30:51.261306 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:51.261310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:51.261323 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:51.261356 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:51.261358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:51.261403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:51.261476 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:51.261879 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.262417 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.263122 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.263330 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.263397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.265848 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:46081: BlockingRecv error: recv got EOF from 127.15.222.65:46081 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:51.457752 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:51.452694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:56457)
0809 03:30:51.452938 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:51.452951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:51.453049 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:51.453259 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:51.453261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:51.453438 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:51.453533 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.453539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.454459 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.454461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.455011 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.455019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.455414 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.455417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.455472 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.455930 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:51.455941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:51.457618 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
W20260809 03:30:51.457921 31279 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:51.452759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:56457 (local address 127.15.222.126:35193)
0809 03:30:51.452891 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:51.452895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:51.453007 (+ 112us) server_negotiation.cc:407] Connection header received
0809 03:30:51.453114 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:51.453116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:51.453165 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:51.453234 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:51.453638 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.454373 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.455105 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.455334 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.455407 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.457800 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:56457: BlockingRecv error: recv got EOF from 127.15.222.66:56457 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:51 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:51.648321 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:51.643181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:43083)
0809 03:30:51.643460 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:51.643475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:51.643571 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:51.643818 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:51.643820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:51.644021 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:51.644126 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.644132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.644813 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.644815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.645398 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:51.645411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.645855 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.645858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:51.645951 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.646404 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:51.646415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:51.648201 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260809 03:30:51.648484 31279 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:51.643248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:43083 (local address 127.15.222.126:35193)
0809 03:30:51.643369 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:51.643372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:51.643538 (+ 166us) server_negotiation.cc:407] Connection header received
0809 03:30:51.643655 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:51.643658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:51.643707 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:51.643768 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:51.644217 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.644735 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.645510 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:51.645749 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:51.645836 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:51.648327 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:43083: BlockingRecv error: recv got EOF from 127.15.222.67:43083 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:52.271588 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:52.266481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:45917)
0809 03:30:52.266759 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:52.266772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:52.266871 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:52.267079 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:52.267081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:52.267256 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:52.267353 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.267358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.268063 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.268065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.268713 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.268724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.269218 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.269222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.269301 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.269752 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:52.269763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:52.271441 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
W20260809 03:30:52.271696 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:52.266550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:45917 (local address 127.15.222.126:35193)
0809 03:30:52.266776 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:52.266781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:52.266835 (+ 54us) server_negotiation.cc:407] Connection header received
0809 03:30:52.266940 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:52.266942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:52.266986 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:52.267058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:52.267448 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.267984 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.268823 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.269101 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.269201 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.271581 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:45917: BlockingRecv error: recv got EOF from 127.15.222.65:45917 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":58,"threads_started":1}
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:52.463246 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:52.458448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:47065)
0809 03:30:52.458768 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:52.458780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:52.458875 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:52.459082 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:52.459084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:52.459259 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:52.459356 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.459361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.460026 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.460028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.460573 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.460580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.460914 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.460916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.460970 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.461418 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:52.461429 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:52.463131 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
W20260809 03:30:52.463361 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:52.458537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:47065 (local address 127.15.222.126:35193)
0809 03:30:52.458669 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:52.458673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:52.458836 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:30:52.458945 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:52.458947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:52.458992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:52.459056 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:52.459447 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.459953 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.460649 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.460845 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.460906 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.463258 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:47065: BlockingRecv error: recv got EOF from 127.15.222.66:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:52 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:52.655622 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:52.649202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:41067)
0809 03:30:52.649445 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:52.649458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:52.649559 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:52.649913 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:52.649932 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:52.650147 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:52.650260 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.650267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.651272 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.651276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.652089 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:52.652099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.652608 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.652612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:52.652694 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.653204 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:52.653218 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:52.655472 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
W20260809 03:30:52.655771 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:52.649459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:41067 (local address 127.15.222.126:35193)
0809 03:30:52.649585 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:52.649588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:52.649600 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:30:52.649635 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:52.649637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:52.649681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:52.649767 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:52.650387 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.651162 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.652200 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:52.652490 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:52.652595 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:52.655643 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:41067: BlockingRecv error: recv got EOF from 127.15.222.67:41067 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:53.277495 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:53.272336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:46683)
0809 03:30:53.272611 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:53.272625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:53.272715 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:53.272939 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:53.272941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:53.273123 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:53.273222 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.273227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.273975 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.273977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.274526 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.274534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.274902 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.274904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.274962 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.275463 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:53.275542 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:53.277377 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260809 03:30:53.277649 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:53.272429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:46683 (local address 127.15.222.126:35193)
0809 03:30:53.272671 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:53.272674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:53.272687 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:53.272795 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:53.272797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:53.272839 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:53.272908 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:53.273304 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.273902 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.274606 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.274815 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.274883 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.277512 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:46683: BlockingRecv error: recv got EOF from 127.15.222.65:46683 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:53.468930 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:53.463952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:55175)
0809 03:30:53.464212 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:53.464226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:53.464319 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:53.464526 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:53.464528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:53.464709 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:53.464806 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.464810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.465507 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.465510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.466173 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.466184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.466570 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.466572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.466633 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.467082 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:53.467092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:53.468790 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
W20260809 03:30:53.469036 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:53.464016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:55175 (local address 127.15.222.126:35193)
0809 03:30:53.464149 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:53.464154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:53.464284 (+ 130us) server_negotiation.cc:407] Connection header received
0809 03:30:53.464399 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:53.464401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:53.464444 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:53.464498 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:53.464892 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.465396 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.466265 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.466479 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.466556 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.468923 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:55175: BlockingRecv error: recv got EOF from 127.15.222.66:55175 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:53 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:53.661610 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:53.656325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:48073)
0809 03:30:53.656588 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:53.656605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:53.656710 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:53.656944 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:53.656947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:53.657146 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:53.657240 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.657246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.658032 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.658034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.658600 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:53.658608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.659034 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.659038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:53.659120 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.659562 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:53.659573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:53.661448 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
W20260809 03:30:53.661665 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:53.656486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:48073 (local address 127.15.222.126:35193)
0809 03:30:53.656623 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:53.656628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:53.656659 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:30:53.656792 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:53.656795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:53.656843 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:53.656919 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:53.657347 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.657922 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.658704 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:53.658930 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:53.659017 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:53.661565 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:48073: BlockingRecv error: recv got EOF from 127.15.222.67:48073 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:54.283260 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:54.278223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:40291)
0809 03:30:54.278534 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:54.278548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:54.278672 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:54.278965 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:54.278968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:54.279165 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:54.279266 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.279273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.279953 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.279955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.280493 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.280499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.280843 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.280845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.280913 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.281382 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:54.281396 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:54.283130 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
W20260809 03:30:54.283353 31295 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:54.278278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:40291 (local address 127.15.222.126:35193)
0809 03:30:54.278542 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:54.278546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:54.278622 (+ 76us) server_negotiation.cc:407] Connection header received
0809 03:30:54.278776 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:54.278779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:54.278830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:54.278931 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:54.279359 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.279869 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.280569 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.280772 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.280834 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.283244 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:40291: BlockingRecv error: recv got EOF from 127.15.222.65:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":61,"threads_started":1}
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:54.474390 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:54.469595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:34323)
0809 03:30:54.469851 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:54.469864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:54.469965 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:54.470168 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:54.470170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:54.470343 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:54.470448 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.470453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.471101 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.471103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.471646 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.471652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.471993 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.471996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.472051 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.472484 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:54.472495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:54.474267 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260809 03:30:54.474519 31295 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:54.469640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34323 (local address 127.15.222.126:35193)
0809 03:30:54.469797 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:54.469801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:54.469926 (+ 125us) server_negotiation.cc:407] Connection header received
0809 03:30:54.470032 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:54.470034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:54.470077 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:54.470140 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:54.470527 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.471030 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.471722 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.471921 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.471984 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.474402 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:34323: BlockingRecv error: recv got EOF from 127.15.222.66:34323 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:54 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:54.667624 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:54.662481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:47255)
0809 03:30:54.662729 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:54.662742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:54.662841 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:54.663043 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:54.663045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:54.663224 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:54.663319 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.663325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.664012 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.664014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.664578 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:54.664586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.664970 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.664973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:54.665050 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.665538 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:54.665549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:54.667468 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
W20260809 03:30:54.667744 31295 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:54.662539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:47255 (local address 127.15.222.126:35193)
0809 03:30:54.662688 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:54.662692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:54.662805 (+ 113us) server_negotiation.cc:407] Connection header received
0809 03:30:54.662911 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:54.662913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:54.662959 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:54.663020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:54.663410 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.663920 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.664663 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:54.664870 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:54.664935 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:54.667639 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:47255: BlockingRecv error: recv got EOF from 127.15.222.67:47255 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:55.289117 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:55.284083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:44127)
0809 03:30:55.284351 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:55.284364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:55.284475 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:55.284684 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:55.284686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:55.284859 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:55.284956 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.284962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.285635 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.285637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.286199 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.286206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.286560 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.286562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.286616 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.287107 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:55.287117 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:55.288954 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260809 03:30:55.289252 31300 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:55.284163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:44127 (local address 127.15.222.126:35193)
0809 03:30:55.284375 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:55.284379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:55.284440 (+ 61us) server_negotiation.cc:407] Connection header received
0809 03:30:55.284544 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:55.284547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:55.284590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:55.284663 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:55.285038 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.285560 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.286273 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.286486 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.286554 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.289119 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:44127: BlockingRecv error: recv got EOF from 127.15.222.65:44127 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:55.480001 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:55.475148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:39625)
0809 03:30:55.475427 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:55.475440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:55.475537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:55.475742 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:55.475744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:55.475937 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:55.476037 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.476042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.476737 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.476739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.477287 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.477293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.477646 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.477648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.477703 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.478172 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:55.478182 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:55.479886 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
W20260809 03:30:55.480108 31300 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:55.475296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:39625 (local address 127.15.222.126:35193)
0809 03:30:55.475436 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:55.475440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:55.475500 (+ 60us) server_negotiation.cc:407] Connection header received
0809 03:30:55.475605 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:55.475607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:55.475649 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:55.475716 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:55.476127 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.476661 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.477366 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.477570 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.477637 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.480010 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:39625: BlockingRecv error: recv got EOF from 127.15.222.66:39625 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:55 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:55.676952 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:55.668373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:33419)
0809 03:30:55.668630 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:55.668643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:55.668760 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:55.669832 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:55.669835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:55.670021 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:55.670122 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.670128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.670796 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.670798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.671353 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:55.671360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.671708 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.671710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:55.671766 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.675029 (+ 3263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:55.675041 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:55.676824 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
W20260809 03:30:55.677073 31300 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:55.668444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:33419 (local address 127.15.222.126:35193)
0809 03:30:55.668595 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:55.668600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:55.668704 (+ 104us) server_negotiation.cc:407] Connection header received
0809 03:30:55.669636 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:55.669639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:55.669688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:55.669785 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:55.670208 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.670716 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.671435 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:55.671635 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:55.671699 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:55.676967 (+ 5268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:33419: BlockingRecv error: recv got EOF from 127.15.222.67:33419 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:56.294941 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:56.289978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:42515)
0809 03:30:56.290250 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:56.290264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:56.290365 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:56.290577 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:56.290579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:56.290775 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:56.290878 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.290884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.291605 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.291608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.292174 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.292183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.292571 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.292573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.292631 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.293091 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:56.293109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:56.294828 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
W20260809 03:30:56.295079 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:56.290040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42515 (local address 127.15.222.126:35193)
0809 03:30:56.290259 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:56.290262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:56.290327 (+ 65us) server_negotiation.cc:407] Connection header received
0809 03:30:56.290433 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:56.290435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:56.290477 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:56.290552 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:56.290965 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.291493 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.292262 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.292489 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.292558 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.294960 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:42515: BlockingRecv error: recv got EOF from 127.15.222.65:42515 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:56.485782 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:56.480690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:48093)
0809 03:30:56.480963 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:56.480979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:56.481090 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:56.481328 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:56.481331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:56.481536 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:56.481629 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.481635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.482382 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.482385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.482946 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.482954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.483381 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.483385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.483463 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.483905 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:56.483915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:56.485630 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260809 03:30:56.485929 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:56.480824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:48093 (local address 127.15.222.126:35193)
0809 03:30:56.480964 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:56.480969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:56.481037 (+ 68us) server_negotiation.cc:407] Connection header received
0809 03:30:56.481179 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:56.481182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:56.481228 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:56.481290 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:56.481760 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.482280 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.483059 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.483282 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.483367 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.485810 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:48093: BlockingRecv error: recv got EOF from 127.15.222.66:48093 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:56 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:56.682794 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:56.677695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:55551)
0809 03:30:56.677984 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:56.677998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:56.678111 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:56.678355 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:56.678357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:56.678586 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:56.678682 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.678689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.679435 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.679438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.679984 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:56.679991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.680426 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.680430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:56.680515 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.680971 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:56.680981 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:56.682642 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":68,"threads_started":1}
W20260809 03:30:56.682888 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:56.677795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:55551 (local address 127.15.222.126:35193)
0809 03:30:56.677942 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:56.677946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:56.678067 (+ 121us) server_negotiation.cc:407] Connection header received
0809 03:30:56.678198 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:56.678201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:56.678253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:56.678319 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:56.678798 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.679325 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.680096 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:56.680318 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:56.680404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:56.682786 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:55551: BlockingRecv error: recv got EOF from 127.15.222.67:55551 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:57.300985 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:57.295627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:55905)
0809 03:30:57.295906 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:57.295920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:57.296009 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:57.296226 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:57.296228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:57.296405 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:57.296500 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.296505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.297193 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.297195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.297771 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.297779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.298147 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.298149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.298206 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.298704 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:57.298716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:57.300838 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260809 03:30:57.301201 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:57.295729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:55905 (local address 127.15.222.126:35193)
0809 03:30:57.296030 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:57.296034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:57.296062 (+ 28us) server_negotiation.cc:407] Connection header received
0809 03:30:57.296095 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:57.296097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:57.296138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:57.296204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:57.296586 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.297118 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.297848 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.298068 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.298136 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.300983 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:55905: BlockingRecv error: recv got EOF from 127.15.222.65:55905 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":64,"threads_started":1}
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:57.491516 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:57.486704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:39715)
0809 03:30:57.487004 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:57.487016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:57.487117 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:57.487325 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:57.487327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:57.487503 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:57.487599 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.487604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.488254 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.488256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.488789 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.488795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.489129 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.489131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.489185 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.489626 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:57.489637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:57.491364 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
W20260809 03:30:57.491617 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:57.486791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:39715 (local address 127.15.222.126:35193)
0809 03:30:57.486941 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:57.486944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:57.487078 (+ 134us) server_negotiation.cc:407] Connection header received
0809 03:30:57.487197 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:57.487199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:57.487241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:57.487298 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:57.487681 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.488180 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.488862 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.489059 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.489121 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.491508 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:39715: BlockingRecv error: recv got EOF from 127.15.222.66:39715 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:57 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.67@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:57.688339 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:57.683466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.67:52851)
0809 03:30:57.683724 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:57.683738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:57.683834 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:57.684044 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:57.684046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:57.684227 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:57.684323 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.684328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.685001 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.685003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.685539 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:57.685545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.685934 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.685936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:57.685997 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.686476 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:57.686487 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:57.688225 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
W20260809 03:30:57.688465 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:57.683530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:52851 (local address 127.15.222.126:35193)
0809 03:30:57.683676 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:57.683680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:57.683798 (+ 118us) server_negotiation.cc:407] Connection header received
0809 03:30:57.683904 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:57.683907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:57.683954 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:57.684017 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:57.684412 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.684924 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.685614 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:57.685848 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:57.685919 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:57.688351 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.67:52851: BlockingRecv error: recv got EOF from 127.15.222.67:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 03:30:58 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:58 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.65@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:58.307780 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:58.301879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.65:53357)
0809 03:30:58.302133 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:58.302146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:58.302228 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:58.302453 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:58.302455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:58.302659 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:58.302786 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:58.302795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.303788 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.303792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:58.304629 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:58.304642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.305233 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.305236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:58.305324 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:58.305903 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:58.305916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:58.307624 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
W20260809 03:30:58.307919 31319 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:58.301942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:53357 (local address 127.15.222.126:35193)
0809 03:30:58.302175 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:58.302178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:58.302191 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:30:58.302298 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:58.302300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:58.302342 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:58.302428 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:58.302936 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.303679 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.304766 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.305126 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.305233 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:58.307770 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.65:53357: BlockingRecv error: recv got EOF from 127.15.222.65:53357 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
Aug 09 03:30:58 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 03:30:58 dist-test-slave-cvwc krb5kdc[29670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.222.66@KRBTEST.COM for oryx/127.15.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 03:30:58.497697 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:58.492449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:35193 (local address 127.15.222.66:43049)
0809 03:30:58.492702 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:58.492717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:58.492825 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:58.493081 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:58.493084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:58.493280 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:58.493374 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:58.493380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.494262 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.494266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:58.494820 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:58.494829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.495262 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.495266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:58.495348 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:58.495785 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:30:58.495795 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:58.497534 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.222.126:35193: Server oryx/127.15.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
W20260809 03:30:58.497803 31319 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:58.492567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:43049 (local address 127.15.222.126:35193)
0809 03:30:58.492705 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:58.492709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:58.492778 (+ 69us) server_negotiation.cc:407] Connection header received
0809 03:30:58.492930 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:58.492933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:58.492982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:58.493042 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:58.493490 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.494158 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.494936 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.495163 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.495248 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:58.497671 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.222.66:43049: BlockingRecv error: recv got EOF from 127.15.222.66:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:30:58.577358 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 30602
I20260809 03:30:58.587721 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 30839
I20260809 03:30:58.604882 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 31038
I20260809 03:30:58.615026 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 30499
2026-08-09T03:30:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18100 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T03:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:58Z Disabled control of system clock
I20260809 03:30:58.651894 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41461
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 03:30:58.761552 31325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:58.761844 31325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:58.761881 31325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:58.765694 31325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:58.765770 31325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:58.765787 31325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:58.765805 31325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:58.765827 31325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:58.770403 31325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:58.771370 31325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:58.772660 31325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:58.780983 31333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:58.781010 31332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:58.781073 31325 server_base.cc:1034] running on GCE node
W20260809 03:30:58.781014 31335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:58.781760 31325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:58.782807 31325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:58.783972 31325 hybrid_clock.cc:648] HybridClock initialized: now 1786246258783958 us; error 26 us; skew 500 ppm
I20260809 03:30:58.785660 31325 webserver.cc:460] Webserver started at http://127.15.222.126:36037/ using document root <none> and password file <none>
I20260809 03:30:58.786223 31325 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:58.786289 31325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:58.786490 31325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:58.788439 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3137005f60e24248abfc84fa42ffca6d"
format_stamp: "Formatted at 2026-08-09 03:30:58 on dist-test-slave-cvwc"
server_key: "7dc1f8912e0bdfcf36bbcb81be5c2fc7"
server_key_iv: "d977ec804dfe803642fe439da22071ac"
server_key_version: "encryptionkey@0"
I20260809 03:30:58.788968 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3137005f60e24248abfc84fa42ffca6d"
format_stamp: "Formatted at 2026-08-09 03:30:58 on dist-test-slave-cvwc"
server_key: "7dc1f8912e0bdfcf36bbcb81be5c2fc7"
server_key_iv: "d977ec804dfe803642fe439da22071ac"
server_key_version: "encryptionkey@0"
I20260809 03:30:58.792317 31325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:30:58.794551 31342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:58.795652 31325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:58.795763 31325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3137005f60e24248abfc84fa42ffca6d"
format_stamp: "Formatted at 2026-08-09 03:30:58 on dist-test-slave-cvwc"
server_key: "7dc1f8912e0bdfcf36bbcb81be5c2fc7"
server_key_iv: "d977ec804dfe803642fe439da22071ac"
server_key_version: "encryptionkey@0"
I20260809 03:30:58.795849 31325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:58.819469 31325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:58.820035 31325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:58.820174 31325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:58.826400 31325 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41461
I20260809 03:30:58.826399 31418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41461 every 8 connection(s)
I20260809 03:30:58.827327 31325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:58.830472 31419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:58.835731 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: Bootstrap starting.
I20260809 03:30:58.837219 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 31325
I20260809 03:30:58.837307 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:58.837538 16249 external_mini_cluster.cc:1428] Setting key 57ebd2bb0421f5e51c91e1ab947605ed
I20260809 03:30:58.838135 31419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:58.838992 31419 log.cc:824] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:58.840829 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: No bootstrap required, opened a new log
I20260809 03:30:58.844157 31419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } }
I20260809 03:30:58.844359 31419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:58.844416 31419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3137005f60e24248abfc84fa42ffca6d, State: Initialized, Role: FOLLOWER
I20260809 03:30:58.844838 31419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } }
I20260809 03:30:58.844976 31419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:58.845058 31419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:58.845160 31419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:58.846195 31419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } }
I20260809 03:30:58.846539 31419 leader_election.cc:302] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3137005f60e24248abfc84fa42ffca6d; no voters:
I20260809 03:30:58.846612 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:58.838674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.15.222.126:41461)
0809 03:30:58.839308 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:58.839330 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:58.839373 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:30:58.840089 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:58.840107 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:58.840425 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:58.840724 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:58.841140 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.842114 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.843016 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:58.843465 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:58.843621 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:58.844365 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:58.844370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:58.844384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:58.844789 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:58.844844 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:58.844863 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:58.844957 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:58.845396 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 03:30:58.845654 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260809 03:30:58.846848 31419 leader_election.cc:288] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:58.847249 31429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:58.847626 31429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [term 1 LEADER]: Becoming Leader. State: Replica: 3137005f60e24248abfc84fa42ffca6d, State: Running, Role: LEADER
I20260809 03:30:58.848044 31419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:58.848229 31429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } }
I20260809 03:30:58.849956 31433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3137005f60e24248abfc84fa42ffca6d. Latest consensus state: current_term: 1 leader_uuid: "3137005f60e24248abfc84fa42ffca6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } } }
I20260809 03:30:58.849967 31428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3137005f60e24248abfc84fa42ffca6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3137005f60e24248abfc84fa42ffca6d" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41461 } } }
I20260809 03:30:58.850180 31428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:58.850211 31433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d [sys.catalog]: This master's current role is: LEADER
W20260809 03:30:58.853590 31455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:30:58.853667 31455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:30:58.853780 31454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:58.854560 31454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:58.859272 31454 catalog_manager.cc:1347] Generated new cluster ID: 4ba3be40f64848d9a5c6a0b7737a3165
I20260809 03:30:58.859331 31454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:58.871346 31454 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:58.872155 31454 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:58.877523 31454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3137005f60e24248abfc84fa42ffca6d: Generated new TSK 0
I20260809 03:30:58.878136 31454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:58.888202 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--builtin_ntp_servers=127.15.222.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 03:30:58.998932 31461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:58.999176 31461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:58.999208 31461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:59.003189 31461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:59.003239 31461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:59.003288 31461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:59.007833 31461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:59.008795 31461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:59.010107 31461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:59.019630 31467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.019781 31461 server_base.cc:1034] running on GCE node
W20260809 03:30:59.019649 31469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.019652 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.020237 31461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:59.020787 31461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:59.021929 31461 hybrid_clock.cc:648] HybridClock initialized: now 1786246259021907 us; error 37 us; skew 500 ppm
I20260809 03:30:59.023638 31461 webserver.cc:460] Webserver started at http://127.15.222.65:45517/ using document root <none> and password file <none>
I20260809 03:30:59.024194 31461 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:59.024252 31461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:59.024448 31461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:59.026443 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "40e0fe458f7c488488cd45f02daecccd"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "e75d5b23348f361e23e38eb54608242a"
server_key_iv: "90f6c3ddc4bc0aa8344edbb5cd283f13"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.026968 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "40e0fe458f7c488488cd45f02daecccd"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "e75d5b23348f361e23e38eb54608242a"
server_key_iv: "90f6c3ddc4bc0aa8344edbb5cd283f13"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.030400 31461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:30:59.032634 31481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.033785 31461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 03:30:59.033902 31461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "40e0fe458f7c488488cd45f02daecccd"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "e75d5b23348f361e23e38eb54608242a"
server_key_iv: "90f6c3ddc4bc0aa8344edbb5cd283f13"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.033991 31461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:59.056823 31461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:59.057428 31461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:59.057590 31461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:59.058146 31461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:59.059047 31461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:59.059094 31461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.059146 31461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:59.059176 31461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.067318 31461 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:46691
I20260809 03:30:59.067348 31641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:46691 every 8 connection(s)
I20260809 03:30:59.068187 31461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:59.074811 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 31461
I20260809 03:30:59.074911 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:59.075173 16249 external_mini_cluster.cc:1428] Setting key cd7771091ea51c3409c9a49f6c220e00
I20260809 03:30:59.077095 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--builtin_ntp_servers=127.15.222.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 03:30:59.077394 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.069954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:60327 (local address 127.15.222.126:41461)
0809 03:30:59.070086 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.070090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.071104 (+ 1014us) server_negotiation.cc:407] Connection header received
0809 03:30:59.072652 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.072655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.072721 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.072792 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.073667 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.074219 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.075394 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.075727 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.075821 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.076911 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:59.076914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:59.076916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:59.076928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:59.076947 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:59.076955 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:59.077010 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:59.077178 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 03:30:59.077246 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:30:59.078415 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.070332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41461 (local address 127.15.222.65:60327)
0809 03:30:59.070964 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:59.071005 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:59.072443 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:59.072981 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:59.072988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:59.073069 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:59.073529 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.073542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.074343 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.074345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.075265 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.075277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.075874 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.075879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.076039 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.076637 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:59.076639 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:59.076689 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:59.076764 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:59.077034 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:59.077042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:59.077048 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:59.077177 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 03:30:59.077453 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260809 03:30:59.079628 31642 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41461
I20260809 03:30:59.079834 31642 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:59.080353 31642 heartbeater.cc:511] Master 127.15.222.126:41461 requested a full tablet report, sending...
I20260809 03:30:59.081832 31368 ts_manager.cc:194] Registered new tserver with Master: 40e0fe458f7c488488cd45f02daecccd (127.15.222.65:46691)
I20260809 03:30:59.082963 31368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.65:60327
W20260809 03:30:59.186810 31647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:59.187048 31647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:59.187098 31647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:59.190989 31647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:59.191035 31647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:59.191102 31647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:30:59.195763 31647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:59.196712 31647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:59.198036 31647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:59.207062 31656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.207032 31653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.207027 31654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.207201 31647 server_base.cc:1034] running on GCE node
I20260809 03:30:59.207557 31647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:59.208092 31647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:59.209236 31647 hybrid_clock.cc:648] HybridClock initialized: now 1786246259209211 us; error 38 us; skew 500 ppm
I20260809 03:30:59.210887 31647 webserver.cc:460] Webserver started at http://127.15.222.66:37049/ using document root <none> and password file <none>
I20260809 03:30:59.211400 31647 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:59.211459 31647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:59.211647 31647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:59.213537 31647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "7b9075a0d1284883880e9f820c383add"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "bfa917c62997520e9012cab45f95629b"
server_key_iv: "63ea84a049eef538cb7fe097642e788a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.214090 31647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "7b9075a0d1284883880e9f820c383add"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "bfa917c62997520e9012cab45f95629b"
server_key_iv: "63ea84a049eef538cb7fe097642e788a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.217471 31647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:30:59.219807 31664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.220878 31647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:30:59.220973 31647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "7b9075a0d1284883880e9f820c383add"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "bfa917c62997520e9012cab45f95629b"
server_key_iv: "63ea84a049eef538cb7fe097642e788a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.221066 31647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:59.231761 31647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:59.232301 31647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:59.232457 31647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:59.232954 31647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:59.233845 31647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:59.233889 31647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.233927 31647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:59.233955 31647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.241952 31647 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:40927
I20260809 03:30:59.241973 31826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:40927 every 8 connection(s)
I20260809 03:30:59.242758 31647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:30:59.244076 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 31647
I20260809 03:30:59.244163 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:30:59.244437 16249 external_mini_cluster.cc:1428] Setting key 95833dec03bd7824ba38e09e75bf48b1
I20260809 03:30:59.246490 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--builtin_ntp_servers=127.15.222.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 03:30:59.251752 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.245817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:43689 (local address 127.15.222.126:41461)
0809 03:30:59.245935 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.245939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.245954 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:30:59.247023 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.247026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.247079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.247150 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.248000 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.248669 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.249966 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.250244 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.250345 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.251288 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:59.251292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:59.251295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:59.251311 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:59.251335 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:59.251346 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:59.251447 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:59.251546 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 03:30:59.251605 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:30:59.252506 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.244752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41461 (local address 127.15.222.66:43689)
0809 03:30:59.245483 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:59.245521 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:59.246825 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:59.247319 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:59.247326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:59.247408 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:59.247860 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.247872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.248787 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.248791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.249849 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.249859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.250383 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.250386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.250531 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.251047 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:59.251048 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:59.251084 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:59.251141 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:59.251437 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:59.251441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:59.251444 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:30:59.251527 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 03:30:59.251762 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":161,"threads_started":1}
I20260809 03:30:59.253572 31827 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41461
I20260809 03:30:59.253813 31827 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:59.254292 31827 heartbeater.cc:511] Master 127.15.222.126:41461 requested a full tablet report, sending...
I20260809 03:30:59.255270 31368 ts_manager.cc:194] Registered new tserver with Master: 7b9075a0d1284883880e9f820c383add (127.15.222.66:40927)
I20260809 03:30:59.255807 31368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.66:43689
W20260809 03:30:59.357549 31834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:59.357833 31834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:59.357906 31834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:59.361826 31834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:59.361877 31834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:59.361956 31834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:30:59.366523 31834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:59.367522 31834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:59.368852 31834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:59.377897 31842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.377964 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.377933 31841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.378254 31834 server_base.cc:1034] running on GCE node
I20260809 03:30:59.378639 31834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:59.379238 31834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:59.380394 31834 hybrid_clock.cc:648] HybridClock initialized: now 1786246259380373 us; error 32 us; skew 500 ppm
I20260809 03:30:59.382176 31834 webserver.cc:460] Webserver started at http://127.15.222.67:39285/ using document root <none> and password file <none>
I20260809 03:30:59.382709 31834 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:59.382768 31834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:59.382982 31834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:59.384935 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "c05f2c739d20466eae0ef0c131f43620"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "fb72d60f039ada7495b6ee71c80697dc"
server_key_iv: "5593e5584cbf28ad932abf9b6136f799"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.385454 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "c05f2c739d20466eae0ef0c131f43620"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "fb72d60f039ada7495b6ee71c80697dc"
server_key_iv: "5593e5584cbf28ad932abf9b6136f799"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.388940 31834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:59.391281 31854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.392416 31834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:59.392519 31834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "c05f2c739d20466eae0ef0c131f43620"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "fb72d60f039ada7495b6ee71c80697dc"
server_key_iv: "5593e5584cbf28ad932abf9b6136f799"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.392607 31834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:59.410828 31834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:59.411458 31834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:59.411609 31834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:59.412148 31834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:59.413028 31834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:59.413075 31834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.413125 31834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:59.413151 31834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.421461 31834 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:41429
I20260809 03:30:59.421487 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:41429 every 8 connection(s)
I20260809 03:30:59.422338 31834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:30:59.423733 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 31834
I20260809 03:30:59.423810 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:30:59.424117 16249 external_mini_cluster.cc:1428] Setting key d158fc2529b0f05ebf9cc45be22cbdf6
I20260809 03:30:59.430692 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.424060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:56707 (local address 127.15.222.126:41461)
0809 03:30:59.424201 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.424206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.425341 (+ 1135us) server_negotiation.cc:407] Connection header received
0809 03:30:59.426724 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.426726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.426771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.426832 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.427603 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.428106 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.429140 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.429345 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.429418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.430330 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:59.430333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:59.430335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:59.430346 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:59.430367 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:59.430374 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:59.430436 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:59.430539 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 03:30:59.430588 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:30:59.431465 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.424472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41461 (local address 127.15.222.67:56707)
0809 03:30:59.425209 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:59.425257 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:59.426551 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:59.426963 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:59.426969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:59.427046 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:59.427505 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.427518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.428188 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.428190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.429055 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:30:59.429064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:30:59.429449 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:30:59.429452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:30:59.429586 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:30:59.430107 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:59.430109 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:59.430145 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:59.430204 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:59.430442 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:59.430447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:59.430451 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:30:59.430532 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:30:59.430735 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260809 03:30:59.432399 32042 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41461
I20260809 03:30:59.432600 32042 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:59.433069 32042 heartbeater.cc:511] Master 127.15.222.126:41461 requested a full tablet report, sending...
I20260809 03:30:59.433976 31368 ts_manager.cc:194] Registered new tserver with Master: c05f2c739d20466eae0ef0c131f43620 (127.15.222.67:41429)
I20260809 03:30:59.434511 31368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.67:56707
I20260809 03:30:59.437644 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:30:59.440325 31425 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:30:59.439448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.15.222.126:41461)
0809 03:30:59.439568 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.439573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.439739 (+ 166us) server_negotiation.cc:407] Connection header received
0809 03:30:59.439848 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.439851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.439894 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.439959 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.440209 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46412: BlockingRecv error: recv got EOF from 127.0.0.1:46412 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:59.440436 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 31461
I20260809 03:30:59.464267 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 31647
I20260809 03:30:59.472410 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 31834
I20260809 03:30:59.487411 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 31325
2026-08-09T03:30:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (867 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T03:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:30:59Z Disabled control of system clock
I20260809 03:30:59.519804 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43485
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:43485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:30:59.626641 32059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:59.626874 32059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:59.626907 32059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:59.630716 32059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:30:59.630757 32059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:59.630771 32059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:59.630785 32059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:30:59.630805 32059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:30:59.635298 32059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:43485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:59.636176 32059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:59.637404 32059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:59.645103 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.645128 32064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.645140 32067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.645265 32059 server_base.cc:1034] running on GCE node
I20260809 03:30:59.645990 32059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:59.646989 32059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:59.648132 32059 hybrid_clock.cc:648] HybridClock initialized: now 1786246259648111 us; error 33 us; skew 500 ppm
I20260809 03:30:59.649827 32059 webserver.cc:460] Webserver started at http://127.15.222.126:42847/ using document root <none> and password file <none>
I20260809 03:30:59.650341 32059 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:59.650403 32059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:59.650602 32059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:59.652489 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "5246356cea74405dbf2b4e160f99e42c"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "ab5bb91ba961aaf3a3c8fad3b2774040"
server_key_iv: "2b8596c04bd19d56cf3107edcfc6499a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.653007 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "5246356cea74405dbf2b4e160f99e42c"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "ab5bb91ba961aaf3a3c8fad3b2774040"
server_key_iv: "2b8596c04bd19d56cf3107edcfc6499a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.656349 32059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:30:59.658537 32082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.659612 32059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:30:59.659720 32059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "5246356cea74405dbf2b4e160f99e42c"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "ab5bb91ba961aaf3a3c8fad3b2774040"
server_key_iv: "2b8596c04bd19d56cf3107edcfc6499a"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.659808 32059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:59.664047 32059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:59.664582 32059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:59.664726 32059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:59.670945 32059 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:43485
I20260809 03:30:59.670961 32172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:43485 every 8 connection(s)
I20260809 03:30:59.671818 32059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:30:59.674247 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 32059
I20260809 03:30:59.674351 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:30:59.674607 16249 external_mini_cluster.cc:1428] Setting key 81719331834b80d989e2d0f9985d6a6a
I20260809 03:30:59.674997 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:30:59.680187 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.676092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.15.222.126:43485)
0809 03:30:59.676718 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.676734 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.676769 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:30:59.677561 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.677578 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.677997 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.678297 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.679032 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:59.679037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:59.679049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:59.679060 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:59.679116 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:59.679132 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:59.679254 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:59.679325 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:30:59.679562 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260809 03:30:59.681342 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c: Bootstrap starting.
I20260809 03:30:59.683629 32173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c: Neither blocks nor log segments found. Creating new log.
I20260809 03:30:59.684288 32173 log.cc:824] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:30:59.686160 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c: No bootstrap required, opened a new log
I20260809 03:30:59.689894 32173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } }
I20260809 03:30:59.690104 32173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:30:59.690153 32173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5246356cea74405dbf2b4e160f99e42c, State: Initialized, Role: FOLLOWER
I20260809 03:30:59.690553 32173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } }
I20260809 03:30:59.690718 32173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:30:59.690796 32173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:30:59.690904 32173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:30:59.691751 32173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } }
I20260809 03:30:59.692080 32173 leader_election.cc:302] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5246356cea74405dbf2b4e160f99e42c; no voters:
I20260809 03:30:59.692355 32173 leader_election.cc:288] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:30:59.692451 32179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:30:59.692651 32179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [term 1 LEADER]: Becoming Leader. State: Replica: 5246356cea74405dbf2b4e160f99e42c, State: Running, Role: LEADER
I20260809 03:30:59.692955 32179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } }
I20260809 03:30:59.693508 32173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:30:59.694509 32183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5246356cea74405dbf2b4e160f99e42c. Latest consensus state: current_term: 1 leader_uuid: "5246356cea74405dbf2b4e160f99e42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } } }
I20260809 03:30:59.694519 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5246356cea74405dbf2b4e160f99e42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5246356cea74405dbf2b4e160f99e42c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43485 } } }
I20260809 03:30:59.694645 32183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:59.694671 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c [sys.catalog]: This master's current role is: LEADER
I20260809 03:30:59.694962 32193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:30:59.698428 32193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:30:59.707042 32193 catalog_manager.cc:1347] Generated new cluster ID: beec8b45858b47e99e392c623af10061
I20260809 03:30:59.707114 32193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:30:59.724045 32193 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:30:59.724403 32193 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:30:59.737272 32193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5246356cea74405dbf2b4e160f99e42c: Generated new TSK 0
I20260809 03:30:59.738107 32193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:30:59.741410 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--builtin_ntp_servers=127.15.222.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:30:59.850183 32209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:30:59.850432 32209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:30:59.850484 32209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:30:59.854369 32209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:30:59.854420 32209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:30:59.854492 32209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:30:59.859061 32209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:30:59.860011 32209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:30:59.861312 32209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:30:59.870235 32216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:30:59.870277 32218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.870332 32209 server_base.cc:1034] running on GCE node
W20260809 03:30:59.870234 32220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:30:59.870807 32209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:30:59.871371 32209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:30:59.872520 32209 hybrid_clock.cc:648] HybridClock initialized: now 1786246259872497 us; error 36 us; skew 500 ppm
I20260809 03:30:59.874189 32209 webserver.cc:460] Webserver started at http://127.15.222.65:44121/ using document root <none> and password file <none>
I20260809 03:30:59.874704 32209 fs_manager.cc:359] Metadata directory not provided
I20260809 03:30:59.874765 32209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:30:59.874964 32209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:30:59.876897 32209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "8ae6c0abaa9243f9b4b488ee703c452f"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "02a5e6ce46daf7c2f9e9fddf5897d11a"
server_key_iv: "c74c206430efaffdabc0b7fbc1a5bcfc"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.877413 32209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "8ae6c0abaa9243f9b4b488ee703c452f"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "02a5e6ce46daf7c2f9e9fddf5897d11a"
server_key_iv: "c74c206430efaffdabc0b7fbc1a5bcfc"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.880825 32209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:30:59.883137 32229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.884207 32209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:30:59.884301 32209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "8ae6c0abaa9243f9b4b488ee703c452f"
format_stamp: "Formatted at 2026-08-09 03:30:59 on dist-test-slave-cvwc"
server_key: "02a5e6ce46daf7c2f9e9fddf5897d11a"
server_key_iv: "c74c206430efaffdabc0b7fbc1a5bcfc"
server_key_version: "encryptionkey@0"
I20260809 03:30:59.884398 32209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:30:59.888633 32209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:30:59.889179 32209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:30:59.889323 32209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:30:59.889829 32209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:30:59.890661 32209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:30:59.890704 32209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.890753 32209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:30:59.890779 32209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:30:59.898911 32209 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:36889
I20260809 03:30:59.898934 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:36889 every 8 connection(s)
I20260809 03:30:59.899703 32209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:30:59.905287 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.901318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:38737 (local address 127.15.222.126:43485)
0809 03:30:59.901446 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:30:59.901449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:30:59.902478 (+ 1029us) server_negotiation.cc:407] Connection header received
0809 03:30:59.903746 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:30:59.903750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:30:59.903796 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:30:59.903870 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:30:59.904861 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:30:59.904864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:30:59.904867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:30:59.904880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:30:59.904908 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:30:59.904916 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:30:59.904991 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:30:59.905112 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 03:30:59.905155 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:30:59.906113 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:30:59.901669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43485 (local address 127.15.222.65:38737)
0809 03:30:59.902333 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0809 03:30:59.902372 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:30:59.903544 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:30:59.904039 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:30:59.904045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:30:59.904128 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:30:59.904616 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:30:59.904617 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:30:59.904655 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:30:59.904719 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:30:59.904983 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:30:59.904991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:30:59.904997 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:30:59.905145 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 03:30:59.905370 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":109,"threads_started":1}
I20260809 03:30:59.907184 32412 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43485
I20260809 03:30:59.907378 32412 heartbeater.cc:464] Registering TS with master...
I20260809 03:30:59.907825 32412 heartbeater.cc:511] Master 127.15.222.126:43485 requested a full tablet report, sending...
I20260809 03:30:59.907918 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 32209
I20260809 03:30:59.908041 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:30:59.908341 16249 external_mini_cluster.cc:1428] Setting key 288fcce46cf0dde8d3c3d7f572bdfb30
I20260809 03:30:59.909358 32110 ts_manager.cc:194] Registered new tserver with Master: 8ae6c0abaa9243f9b4b488ee703c452f (127.15.222.65:36889)
I20260809 03:30:59.910542 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--builtin_ntp_servers=127.15.222.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 03:31:00.022583 32416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:00.022845 32416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:00.022907 32416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:00.026798 32416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:00.026851 32416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:00.026934 32416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:31:00.031494 32416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:00.032465 32416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:00.033812 32416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:00.042697 32422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:00.042697 32424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:00.042724 32421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:00.042862 32416 server_base.cc:1034] running on GCE node
I20260809 03:31:00.043365 32416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:00.043977 32416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:00.045145 32416 hybrid_clock.cc:648] HybridClock initialized: now 1786246260045128 us; error 27 us; skew 500 ppm
I20260809 03:31:00.046772 32416 webserver.cc:460] Webserver started at http://127.15.222.66:33269/ using document root <none> and password file <none>
I20260809 03:31:00.047276 32416 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:00.047331 32416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:00.047493 32416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:00.049364 32416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "33961a0228724ef3a7a4fbe3b29f1d2e"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "a304670a09b473ce9541e2d0f016932e"
server_key_iv: "2a7d22065cdd39521e886e69c85e070c"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.049901 32416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "33961a0228724ef3a7a4fbe3b29f1d2e"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "a304670a09b473ce9541e2d0f016932e"
server_key_iv: "2a7d22065cdd39521e886e69c85e070c"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.053191 32416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 03:31:00.055505 32431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.056602 32416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:31:00.056699 32416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "33961a0228724ef3a7a4fbe3b29f1d2e"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "a304670a09b473ce9541e2d0f016932e"
server_key_iv: "2a7d22065cdd39521e886e69c85e070c"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.056794 32416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:00.061043 32416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:00.061587 32416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:00.061769 32416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:00.062297 32416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:00.063187 32416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:00.063233 32416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.063292 32416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:00.063323 32416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.071425 32416 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:33813
I20260809 03:31:00.071461 32608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:33813 every 8 connection(s)
I20260809 03:31:00.072258 32416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:00.076876 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 32416
I20260809 03:31:00.076977 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:00.077250 16249 external_mini_cluster.cc:1428] Setting key 892e4d20239e59e4bf6bc8fada3cb904
I20260809 03:31:00.078136 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:00.073929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:52555 (local address 127.15.222.126:43485)
0809 03:31:00.074056 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:00.074060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:00.075158 (+ 1098us) server_negotiation.cc:407] Connection header received
0809 03:31:00.076451 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:00.076455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:00.076503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:00.076581 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:00.077647 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:00.077651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:00.077654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:00.077668 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:00.077686 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:00.077694 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:00.077804 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:00.077956 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 03:31:00.078004 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:31:00.079027 32612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:00.074315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43485 (local address 127.15.222.66:52555)
0809 03:31:00.074996 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:00.075036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:00.076227 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:00.076762 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:00.076769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:00.076854 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:00.077386 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:00.077387 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:00.077427 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:00.077494 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:00.077798 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:00.077806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:00.077812 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:31:00.077978 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 03:31:00.078210 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":109,"threads_started":1}
I20260809 03:31:00.079306 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--builtin_ntp_servers=127.15.222.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 03:31:00.080142 32609 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43485
I20260809 03:31:00.080345 32609 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:00.080708 32609 heartbeater.cc:511] Master 127.15.222.126:43485 requested a full tablet report, sending...
I20260809 03:31:00.081638 32110 ts_manager.cc:194] Registered new tserver with Master: 33961a0228724ef3a7a4fbe3b29f1d2e (127.15.222.66:33813)
W20260809 03:31:00.187431 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:00.187650 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:00.187685 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:00.191529 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:00.191572 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:00.191622 32615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:31:00.196244 32615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:00.197113 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:00.198374 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:00.207193 32624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:00.207203 32625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:00.207254 32615 server_base.cc:1034] running on GCE node
W20260809 03:31:00.207203 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:00.207749 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:00.208320 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:00.209467 32615 hybrid_clock.cc:648] HybridClock initialized: now 1786246260209442 us; error 38 us; skew 500 ppm
I20260809 03:31:00.211102 32615 webserver.cc:460] Webserver started at http://127.15.222.67:43449/ using document root <none> and password file <none>
I20260809 03:31:00.211637 32615 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:00.211699 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:00.211886 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:00.213917 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "dcdab541a2fe4aa59b1cdefae0176f2a"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "af228b17cbfc67d0664116e9ea26c765"
server_key_iv: "bab258ccbee2f03cb41c381592dd4e75"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.214445 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "dcdab541a2fe4aa59b1cdefae0176f2a"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "af228b17cbfc67d0664116e9ea26c765"
server_key_iv: "bab258ccbee2f03cb41c381592dd4e75"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.217909 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:31:00.220184 32638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.221328 32615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:00.221462 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "dcdab541a2fe4aa59b1cdefae0176f2a"
format_stamp: "Formatted at 2026-08-09 03:31:00 on dist-test-slave-cvwc"
server_key: "af228b17cbfc67d0664116e9ea26c765"
server_key_iv: "bab258ccbee2f03cb41c381592dd4e75"
server_key_version: "encryptionkey@0"
I20260809 03:31:00.221553 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:00.225781 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:00.226334 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:00.226485 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:00.226969 32615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:00.227793 32615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:00.227842 32615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.227890 32615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:00.227918 32615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:00.235857 32615 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:42671
I20260809 03:31:00.235877 329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:42671 every 8 connection(s)
I20260809 03:31:00.236629 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:00.242172 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:00.238336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:44571 (local address 127.15.222.126:43485)
0809 03:31:00.238480 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:00.238484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:00.239427 (+ 943us) server_negotiation.cc:407] Connection header received
0809 03:31:00.240641 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:00.240644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:00.240688 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:00.240762 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:00.241736 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:00.241740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:00.241743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:31:00.241756 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:00.241778 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:31:00.241787 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:00.241865 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:00.241998 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 03:31:00.242042 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:31:00.242930 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:00.238677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43485 (local address 127.15.222.67:44571)
0809 03:31:00.239287 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:00.239324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:00.240449 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:00.240935 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:00.240941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:00.241024 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:00.241479 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:31:00.241481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:00.241516 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:31:00.241578 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:00.241858 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:00.241866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:00.241873 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:31:00.242029 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 03:31:00.242252 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260809 03:31:00.243902 331 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43485
I20260809 03:31:00.244100 331 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:00.244463 331 heartbeater.cc:511] Master 127.15.222.126:43485 requested a full tablet report, sending...
I20260809 03:31:00.245378 32110 ts_manager.cc:194] Registered new tserver with Master: dcdab541a2fe4aa59b1cdefae0176f2a (127.15.222.67:42671)
I20260809 03:31:00.245541 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 32615
I20260809 03:31:00.245630 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:00.245962 16249 external_mini_cluster.cc:1428] Setting key 8508a13de1d64dfa4c6b3cc3c00ced4f
I20260809 03:31:00.247723 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 03:31:00.250240 32176 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:31:00.249458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.15.222.126:43485)
0809 03:31:00.249577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:00.249581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:00.249748 (+ 167us) server_negotiation.cc:407] Connection header received
0809 03:31:00.249861 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:00.249863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:00.249898 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:00.249958 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:00.250137 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39580: BlockingRecv error: recv got EOF from 127.0.0.1:39580 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:31:00.250397 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 32209
I20260809 03:31:00.275748 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 32416
I20260809 03:31:00.290639 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 32615
I20260809 03:31:00.309798 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 32059
2026-08-09T03:31:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (819 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:00 dist-test-slave-cvwc krb5kdc[346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:00 dist-test-slave-cvwc krb5kdc[346](info): set up 2 sockets
Aug 09 03:31:00 dist-test-slave-cvwc krb5kdc[346](info): commencing operation
krb5kdc: starting...
W20260809 03:31:01.441831 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:31:01 dist-test-slave-cvwc krb5kdc[346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:31:01.554524 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37985
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:01.661522 362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.661784 362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.661836 362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.665688 362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:01.665755 362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.665786 362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.665814 362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:01.665841 362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:01.670802 362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:01.671711 362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.672652 362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.680492 374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.680501 371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.680508 370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.680536 362 server_base.cc:1034] running on GCE node
I20260809 03:31:01.681303 362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.682319 362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.683471 362 hybrid_clock.cc:648] HybridClock initialized: now 1786246261683445 us; error 38 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-cvwc krb5kdc[346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.689071 362 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:31:01.689968 362 webserver.cc:460] Webserver started at http://127.15.222.126:42053/ using document root <none> and password file <none>
I20260809 03:31:01.690503 362 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.690569 362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.690765 362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:01.692759 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "cb05c40437014e3bb73cf1ca76088c37"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "f089d7385cbad8b1481d0d47f750bf37"
server_key_iv: "fd3b2d2c4ca60e990bd22fa45519e99c"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.693284 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "cb05c40437014e3bb73cf1ca76088c37"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "f089d7385cbad8b1481d0d47f750bf37"
server_key_iv: "fd3b2d2c4ca60e990bd22fa45519e99c"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.696775 362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:31:01.699086 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.700203 362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:01.700312 362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "cb05c40437014e3bb73cf1ca76088c37"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "f089d7385cbad8b1481d0d47f750bf37"
server_key_iv: "fd3b2d2c4ca60e990bd22fa45519e99c"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.700412 362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.715206 362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.717686 362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.717864 362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.724479 362 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:37985
I20260809 03:31:01.724498 466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:37985 every 8 connection(s)
I20260809 03:31:01.725346 362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:01.727949 467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:01.729622 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 362
I20260809 03:31:01.729763 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:31:01.730046 16249 external_mini_cluster.cc:1428] Setting key daa3fd127690f29b6237276ddd7a951d
I20260809 03:31:01.734059 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: Bootstrap starting.
I20260809 03:31:01.736193 467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:01.736958 467 log.cc:824] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:31:01 dist-test-slave-cvwc krb5kdc[346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:01.738970 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: No bootstrap required, opened a new log
I20260809 03:31:01.742478 467 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } }
I20260809 03:31:01.742693 467 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:01.742731 467 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb05c40437014e3bb73cf1ca76088c37, State: Initialized, Role: FOLLOWER
I20260809 03:31:01.743176 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:01.731437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.15.222.126:37985)
0809 03:31:01.731857 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:01.731872 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:01.731915 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:31:01.732525 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:01.732541 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:01.732843 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:01.733146 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:01.733943 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.734923 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.735730 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:01.736187 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:01.736330 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:01.738867 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:01.738872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:01.738887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:01.738915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:01.740452 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.740848 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.740853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.740858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.740969 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:01.741223 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:01.741226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:01.741228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:01.741608 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:01.741824 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:01.742118 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 03:31:01.742392 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260809 03:31:01.743155 467 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } }
I20260809 03:31:01.743269 467 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:01.743321 467 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:01.743414 467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:01.744318 467 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } }
I20260809 03:31:01.744670 467 leader_election.cc:302] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb05c40437014e3bb73cf1ca76088c37; no voters:
I20260809 03:31:01.744997 467 leader_election.cc:288] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:01.745076 472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:01.745278 472 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [term 1 LEADER]: Becoming Leader. State: Replica: cb05c40437014e3bb73cf1ca76088c37, State: Running, Role: LEADER
I20260809 03:31:01.745594 472 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } }
I20260809 03:31:01.746167 467 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:01.746632 472 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb05c40437014e3bb73cf1ca76088c37. Latest consensus state: current_term: 1 leader_uuid: "cb05c40437014e3bb73cf1ca76088c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } } }
I20260809 03:31:01.746830 472 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:01.747113 473 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb05c40437014e3bb73cf1ca76088c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb05c40437014e3bb73cf1ca76088c37" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37985 } } }
I20260809 03:31:01.747203 473 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37 [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:01.748446 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:01.751080 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:01.756405 487 catalog_manager.cc:1347] Generated new cluster ID: 53cc30e46f374f969db5089e07852888
I20260809 03:31:01.756470 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:01.769695 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:01.770918 487 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:01.786550 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: Generated new TSK 0
I20260809 03:31:01.787413 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:31:01.816390 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:01.924562 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:01.924790 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:01.924834 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:01.928707 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:01.928756 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:01.928818 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:31:01.933902 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:01.934844 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:01.935804 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:01.944837 515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.944849 518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:01.944849 514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:01.945041 506 server_base.cc:1034] running on GCE node
I20260809 03:31:01.945495 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:01.946120 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:01.947271 506 hybrid_clock.cc:648] HybridClock initialized: now 1786246261947253 us; error 27 us; skew 500 ppm
Aug 09 03:31:01 dist-test-slave-cvwc krb5kdc[346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:01.949983 506 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:31:01.950789 506 webserver.cc:460] Webserver started at http://127.15.222.65:40685/ using document root <none> and password file <none>
I20260809 03:31:01.951327 506 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:01.951399 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:01.951589 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:01.953480 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "3f4ca269c7164872bbfe60ef259f3a92"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "97c1a5ebb99a62d37a7866a13b5c3988"
server_key_iv: "0eab3c06d6edf71e2636b3054a5d2a10"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.954021 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "3f4ca269c7164872bbfe60ef259f3a92"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "97c1a5ebb99a62d37a7866a13b5c3988"
server_key_iv: "0eab3c06d6edf71e2636b3054a5d2a10"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.957353 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:31:01.959610 527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:01.960744 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:31:01.960848 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "3f4ca269c7164872bbfe60ef259f3a92"
format_stamp: "Formatted at 2026-08-09 03:31:01 on dist-test-slave-cvwc"
server_key: "97c1a5ebb99a62d37a7866a13b5c3988"
server_key_iv: "0eab3c06d6edf71e2636b3054a5d2a10"
server_key_version: "encryptionkey@0"
I20260809 03:31:01.960947 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:01.995618 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:01.998760 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:01.998945 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:01.999526 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:02.000352 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:02.000401 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.000456 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:02.000484 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.008755 506 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:35539
I20260809 03:31:02.008785 697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:35539 every 8 connection(s)
I20260809 03:31:02.009613 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:02.012195 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 506
I20260809 03:31:02.012320 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:02.012646 16249 external_mini_cluster.cc:1428] Setting key bdeb8fc193b048f950524c8b117613a2
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246261, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:31:02.023214 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.011263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:41783 (local address 127.15.222.126:37985)
0809 03:31:02.011377 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.011391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.012219 (+ 828us) server_negotiation.cc:407] Connection header received
0809 03:31:02.013797 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.013800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.013847 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.013935 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:02.015371 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.015898 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.017053 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.017372 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.017472 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.019766 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:02.019771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:02.019773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:02.019803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:02.021319 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.021797 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.021801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.021803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.021913 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.022213 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.022217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.022219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.022438 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:02.022532 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.022983 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.023088 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:31:02.024089 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.011597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37985 (local address 127.15.222.65:41783)
0809 03:31:02.012039 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.012097 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.013597 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.014099 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.014108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.014594 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:02.015219 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.015237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.016032 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.016036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.016939 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.016949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.017495 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.017497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.017627 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.018167 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:02.018183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:02.019604 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:02.021426 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.021431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.021442 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.021673 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.021993 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.021995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.021996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.022115 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.022521 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:02.022525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:02.022742 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.022890 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.023121 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:31:02.025173 699 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37985
I20260809 03:31:02.025411 699 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:02.025974 699 heartbeater.cc:511] Master 127.15.222.126:37985 requested a full tablet report, sending...
I20260809 03:31:02.027697 419 ts_manager.cc:194] Registered new tserver with Master: 3f4ca269c7164872bbfe60ef259f3a92 (127.15.222.65:35539)
I20260809 03:31:02.028790 419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:41783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:31:02.040126 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:02.147584 707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:02.147809 707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:02.147855 707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:02.151793 707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:02.151842 707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:02.151914 707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:31:02.156944 707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:02.157934 707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:02.158910 707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:31:02.168205 707 server_base.cc:1034] running on GCE node
W20260809 03:31:02.168182 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:02.168147 714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:02.168147 713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:02.168758 707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:02.169327 707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:02.170468 707 hybrid_clock.cc:648] HybridClock initialized: now 1786246262170438 us; error 41 us; skew 500 ppm
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246262, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:02.172989 707 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:31:02.173892 707 webserver.cc:460] Webserver started at http://127.15.222.66:33243/ using document root <none> and password file <none>
I20260809 03:31:02.174392 707 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:02.174446 707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:02.174609 707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:02.176586 707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "c844ad257bd04963a139ee84c3e43929"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "ddf02e56b8656076f5f49bca968bbc0f"
server_key_iv: "0c7d0a1cd96ca780c82cb98f79bdba40"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.177109 707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "c844ad257bd04963a139ee84c3e43929"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "ddf02e56b8656076f5f49bca968bbc0f"
server_key_iv: "0c7d0a1cd96ca780c82cb98f79bdba40"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.180604 707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:31:02.182936 733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.184069 707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:31:02.184175 707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "c844ad257bd04963a139ee84c3e43929"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "ddf02e56b8656076f5f49bca968bbc0f"
server_key_iv: "0c7d0a1cd96ca780c82cb98f79bdba40"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.184267 707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:02.196969 707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:02.199848 707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:02.200006 707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:02.200541 707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:02.201298 707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:02.201342 707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.201393 707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:02.201422 707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.209647 707 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:44575
I20260809 03:31:02.209671 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:44575 every 8 connection(s)
I20260809 03:31:02.210467 707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:02.215292 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 707
I20260809 03:31:02.215401 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:02.215655 16249 external_mini_cluster.cc:1428] Setting key f7da047c924f4a5cdfdeb1e0bca19625
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246262, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:02.223659 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.212058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:55849 (local address 127.15.222.126:37985)
0809 03:31:02.212188 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.212192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.213008 (+ 816us) server_negotiation.cc:407] Connection header received
0809 03:31:02.214336 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.214339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.214390 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.214459 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:02.215557 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.216167 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.217514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.217785 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.217876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.220346 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:02.220349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:02.220351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:02.220378 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:02.221706 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.222182 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.222187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.222188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.222294 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.222597 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.222601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.222602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.222822 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:02.222923 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.223335 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.223467 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:31:02.224277 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.212446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37985 (local address 127.15.222.66:55849)
0809 03:31:02.212860 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.212908 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.214160 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.214599 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.214605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.214939 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:02.215413 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.215431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.216296 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.216299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.217394 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.217405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.217927 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.217931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.218084 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.218582 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:02.218597 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:02.220199 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:02.221822 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.221826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.221837 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.222074 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.222392 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.222395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.222396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.222518 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.222908 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:02.222913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:02.223116 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.223260 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.223495 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260809 03:31:02.225346 906 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37985
I20260809 03:31:02.225561 906 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:02.226033 906 heartbeater.cc:511] Master 127.15.222.126:37985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:31:02.226998 419 ts_manager.cc:194] Registered new tserver with Master: c844ad257bd04963a139ee84c3e43929 (127.15.222.66:44575)
I20260809 03:31:02.227541 419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:55849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:31:02.242286 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:02.353003 913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:02.353240 913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:02.353278 913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:02.357089 913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:02.357132 913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:02.357204 913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:31:02.362142 913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:02.363010 913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:02.363924 913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:02.372870 920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:02.372886 919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:02.372922 924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:02.373106 913 server_base.cc:1034] running on GCE node
I20260809 03:31:02.373533 913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:02.374136 913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:02.375281 913 hybrid_clock.cc:648] HybridClock initialized: now 1786246262375266 us; error 27 us; skew 500 ppm
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246262, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:02.380003 913 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:31:02.380836 913 webserver.cc:460] Webserver started at http://127.15.222.67:38617/ using document root <none> and password file <none>
I20260809 03:31:02.381358 913 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:02.381424 913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:02.381623 913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:31:02.383597 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "417075d591f04f0c99c3f97ab44350fa"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "331299f6ac0ff82f3b8bcb22a3d7ebcb"
server_key_iv: "711ac6557fc36e3a20e36ec0601e0334"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.384140 913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "417075d591f04f0c99c3f97ab44350fa"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "331299f6ac0ff82f3b8bcb22a3d7ebcb"
server_key_iv: "711ac6557fc36e3a20e36ec0601e0334"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.387753 913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:31:02.390031 935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.391136 913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:31:02.391238 913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "417075d591f04f0c99c3f97ab44350fa"
format_stamp: "Formatted at 2026-08-09 03:31:02 on dist-test-slave-cvwc"
server_key: "331299f6ac0ff82f3b8bcb22a3d7ebcb"
server_key_iv: "711ac6557fc36e3a20e36ec0601e0334"
server_key_version: "encryptionkey@0"
I20260809 03:31:02.391322 913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:02.406832 913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:02.409274 913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:02.409430 913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:02.409976 913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:02.410728 913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:02.410773 913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.410823 913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:02.410852 913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:02.418856 913 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:43333
I20260809 03:31:02.418877 1107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:43333 every 8 connection(s)
I20260809 03:31:02.419659 913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246262, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:02.428306 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 913
I20260809 03:31:02.428421 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:02.428673 16249 external_mini_cluster.cc:1428] Setting key 1938b3dc8625d20511a1e10889fdc1e1
I20260809 03:31:02.432250 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.421255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:56131 (local address 127.15.222.126:37985)
0809 03:31:02.421411 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.421416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.422219 (+ 803us) server_negotiation.cc:407] Connection header received
0809 03:31:02.423520 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.423522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.423569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.423639 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:02.424758 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.425287 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.426381 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.426591 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.426664 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.428901 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:02.428905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:02.428908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:02.428938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:02.430325 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.430853 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.430857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.430858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.430949 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.431255 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.431258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.431260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.431435 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:02.431520 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.431971 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.432091 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:31:02.432921 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.421621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37985 (local address 127.15.222.67:56131)
0809 03:31:02.422085 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.422124 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.423343 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.423777 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.423783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.424188 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:02.424634 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.424648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.425391 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.425394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.426290 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.426300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.426696 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.426699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.426833 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.427325 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:02.427340 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:02.428715 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:02.430441 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.430446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.430456 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.430766 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.431040 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:02.431043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:02.431045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:02.431173 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:02.431532 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:02.431538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:02.431752 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.431906 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.432187 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260809 03:31:02.433846 1110 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37985
I20260809 03:31:02.434053 1110 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:02.434476 1110 heartbeater.cc:511] Master 127.15.222.126:37985 requested a full tablet report, sending...
I20260809 03:31:02.435365 419 ts_manager.cc:194] Registered new tserver with Master: 417075d591f04f0c99c3f97ab44350fa (127.15.222.67:43333)
I20260809 03:31:02.435909 419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:56131
I20260809 03:31:02.442227 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:02.450253 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.444017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.15.222.126:37985)
0809 03:31:02.444126 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.444129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.444312 (+ 183us) server_negotiation.cc:407] Connection header received
0809 03:31:02.444401 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.444403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.444438 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.444496 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:02.445069 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.445546 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.446294 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.446557 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.446626 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.447784 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:02.447787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:02.447789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:02.447810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:02.449037 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.449353 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.449355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.449356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.449419 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:02.449592 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:02.449593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:02.449594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:02.449820 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:02.449924 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.450059 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.450146 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:31:02.452618 419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:02.455495 419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:02.468802 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.463059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35539 (local address 127.0.0.1:49658)
0809 03:31:02.463710 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.463756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.463892 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.464673 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.464679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.464703 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:02.464790 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.464794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.466038 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.466041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.467317 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.467327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.468386 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.468389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.468477 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.468501 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.468549 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.468662 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":64,"thread_start_us":84,"threads_started":1}
I20260809 03:31:02.468807 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.463512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.15.222.66:44575)
0809 03:31:02.464524 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.464536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.464558 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:31:02.464628 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.464632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.464766 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.464886 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:02.465119 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.466268 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.467636 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.468162 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.468259 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.468418 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.468559 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.468657 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":121,"threads_started":1}
I20260809 03:31:02.469062 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.463199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.15.222.65:35539)
0809 03:31:02.463934 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.463948 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.463971 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:31:02.464043 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.464048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.464285 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.464485 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:02.464907 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.465925 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.467443 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.468246 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.468637 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.468770 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.468816 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.468914 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":63,"threads_started":1}
I20260809 03:31:02.469280 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.463348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:44575 (local address 127.0.0.1:47470)
0809 03:31:02.464255 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.464267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.464350 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.464883 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.464886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.464905 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:02.464992 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.464998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.466417 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.466421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.467526 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.467534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.468272 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.468275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.468378 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.468415 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.468518 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.469154 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":24,"thread_start_us":113,"threads_started":1}
I20260809 03:31:02.470454 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.463776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43333 (local address 127.0.0.1:41286)
0809 03:31:02.464983 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.464995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.465077 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.465510 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.465513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.465533 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:02.465621 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.465627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.467202 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.467205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.468279 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.468290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.470094 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.470097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.470180 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.470203 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.470268 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.470333 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":90,"threads_started":1}
I20260809 03:31:02.470800 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.464176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.15.222.67:43333)
0809 03:31:02.464610 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.464622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.465069 (+ 447us) server_negotiation.cc:407] Connection header received
0809 03:31:02.465180 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.465185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.465370 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.465514 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:02.465754 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.467072 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.468413 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.469787 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.469887 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.470033 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.470606 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.470687 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":130,"threads_started":1}
I20260809 03:31:02.472136 598 tablet_service.cc:1467] Processing CreateTablet for tablet 20f7ca541a994ecc967c78b50092f215 (DEFAULT_TABLE table=test-table [id=ad531fefd61f4d4eb0f1f04ce3ddfc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:02.472136 800 tablet_service.cc:1467] Processing CreateTablet for tablet 20f7ca541a994ecc967c78b50092f215 (DEFAULT_TABLE table=test-table [id=ad531fefd61f4d4eb0f1f04ce3ddfc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:02.473248 800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f7ca541a994ecc967c78b50092f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:02.473248 598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f7ca541a994ecc967c78b50092f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:02.476223 1009 tablet_service.cc:1467] Processing CreateTablet for tablet 20f7ca541a994ecc967c78b50092f215 (DEFAULT_TABLE table=test-table [id=ad531fefd61f4d4eb0f1f04ce3ddfc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:02.477082 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f7ca541a994ecc967c78b50092f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:02.480480 1133 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Bootstrap starting.
I20260809 03:31:02.481235 1136 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Bootstrap starting.
I20260809 03:31:02.482916 1136 tablet_bootstrap.cc:654] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:02.483444 1133 tablet_bootstrap.cc:654] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:02.483877 1137 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Bootstrap starting.
I20260809 03:31:02.484066 1136 log.cc:824] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:02.484668 1133 log.cc:824] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:02.486126 1136 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: No bootstrap required, opened a new log
I20260809 03:31:02.486287 1137 tablet_bootstrap.cc:654] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:02.486330 1136 ts_tablet_manager.cc:1397] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:31:02.486337 1133 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: No bootstrap required, opened a new log
I20260809 03:31:02.486497 1133 ts_tablet_manager.cc:1397] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:31:02.487460 1137 log.cc:824] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:02.489403 1137 tablet_bootstrap.cc:492] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: No bootstrap required, opened a new log
I20260809 03:31:02.489574 1137 ts_tablet_manager.cc:1397] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:31:02.489457 1133 raft_consensus.cc:348] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.489728 1133 raft_consensus.cc:374] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:02.489794 1133 raft_consensus.cc:729] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c844ad257bd04963a139ee84c3e43929, State: Initialized, Role: FOLLOWER
I20260809 03:31:02.490008 1136 raft_consensus.cc:348] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.490252 1136 raft_consensus.cc:374] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:02.490213 1133 consensus_queue.cc:260] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.490319 1136 raft_consensus.cc:729] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4ca269c7164872bbfe60ef259f3a92, State: Initialized, Role: FOLLOWER
I20260809 03:31:02.490721 1136 consensus_queue.cc:260] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.490859 906 heartbeater.cc:503] Master 127.15.222.126:37985 was elected leader, sending a full tablet report...
I20260809 03:31:02.491158 1133 ts_tablet_manager.cc:1428] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:31:02.491606 1136 ts_tablet_manager.cc:1428] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:31:02.491914 699 heartbeater.cc:503] Master 127.15.222.126:37985 was elected leader, sending a full tablet report...
I20260809 03:31:02.493945 1137 raft_consensus.cc:348] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.494196 1137 raft_consensus.cc:374] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:02.494256 1137 raft_consensus.cc:729] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417075d591f04f0c99c3f97ab44350fa, State: Initialized, Role: FOLLOWER
I20260809 03:31:02.494644 1137 consensus_queue.cc:260] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.495501 1110 heartbeater.cc:503] Master 127.15.222.126:37985 was elected leader, sending a full tablet report...
I20260809 03:31:02.495597 1137 ts_tablet_manager.cc:1428] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260809 03:31:02.510710 700 tablet.cc:2369] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:31:02.670608 1113 tablet.cc:2369] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:31:02.711598 907 tablet.cc:2369] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:02.735904 1143 raft_consensus.cc:482] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:02.736089 1143 raft_consensus.cc:504] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.737159 1143 leader_election.cc:288] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4ca269c7164872bbfe60ef259f3a92 (127.15.222.65:35539), 417075d591f04f0c99c3f97ab44350fa (127.15.222.67:43333)
I20260809 03:31:02.740935 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.737471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:60399 (local address 127.15.222.65:35539)
0809 03:31:02.737625 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.737630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.737808 (+ 178us) server_negotiation.cc:407] Connection header received
0809 03:31:02.737877 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.737880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.737943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.738030 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:02.738351 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.738944 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740017 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.740410 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.740536 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.740739 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.740811 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:31:02.740945 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.737367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35539 (local address 127.15.222.66:60399)
0809 03:31:02.737603 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.737616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.737748 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.738060 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.738064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.738087 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:02.738229 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.738236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.739052 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.739056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.739901 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.739911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.740523 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.740617 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.740645 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.740717 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.740783 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260809 03:31:02.740989 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.737888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:57001 (local address 127.15.222.67:43333)
0809 03:31:02.738023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.738028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.738126 (+ 98us) server_negotiation.cc:407] Connection header received
0809 03:31:02.738224 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.738227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.738278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.738344 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:02.738614 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.739120 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740106 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.740519 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740628 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.740692 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.740822 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.740885 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:31:02.740990 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.737757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43333 (local address 127.15.222.66:57001)
0809 03:31:02.738046 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.738058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:02.738152 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:02.738367 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:02.738369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:02.738385 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:02.738519 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.738526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.739217 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.739220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.739981 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:02.739991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.740622 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.740625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:02.740706 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.740729 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:31:02.740800 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:31:02.740861 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":66,"threads_started":1}
I20260809 03:31:02.741626 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f7ca541a994ecc967c78b50092f215" candidate_uuid: "c844ad257bd04963a139ee84c3e43929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417075d591f04f0c99c3f97ab44350fa" is_pre_election: true
I20260809 03:31:02.741768 628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f7ca541a994ecc967c78b50092f215" candidate_uuid: "c844ad257bd04963a139ee84c3e43929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4ca269c7164872bbfe60ef259f3a92" is_pre_election: true
I20260809 03:31:02.741958 1043 raft_consensus.cc:2454] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c844ad257bd04963a139ee84c3e43929 in term 0.
I20260809 03:31:02.742050 628 raft_consensus.cc:2454] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c844ad257bd04963a139ee84c3e43929 in term 0.
I20260809 03:31:02.742413 740 leader_election.cc:302] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417075d591f04f0c99c3f97ab44350fa, c844ad257bd04963a139ee84c3e43929; no voters:
I20260809 03:31:02.742633 1143 raft_consensus.cc:2781] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:02.742708 1143 raft_consensus.cc:482] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:02.742758 1143 raft_consensus.cc:3037] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:02.743798 1143 raft_consensus.cc:504] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.744094 1143 leader_election.cc:288] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [CANDIDATE]: Term 1 election: Requested vote from peers 3f4ca269c7164872bbfe60ef259f3a92 (127.15.222.65:35539), 417075d591f04f0c99c3f97ab44350fa (127.15.222.67:43333)
I20260809 03:31:02.744413 628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f7ca541a994ecc967c78b50092f215" candidate_uuid: "c844ad257bd04963a139ee84c3e43929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4ca269c7164872bbfe60ef259f3a92"
I20260809 03:31:02.744441 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f7ca541a994ecc967c78b50092f215" candidate_uuid: "c844ad257bd04963a139ee84c3e43929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417075d591f04f0c99c3f97ab44350fa"
I20260809 03:31:02.744527 628 raft_consensus.cc:3037] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:02.744539 1043 raft_consensus.cc:3037] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:02.745494 628 raft_consensus.cc:2454] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c844ad257bd04963a139ee84c3e43929 in term 1.
I20260809 03:31:02.745555 1043 raft_consensus.cc:2454] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c844ad257bd04963a139ee84c3e43929 in term 1.
I20260809 03:31:02.745766 740 leader_election.cc:302] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4ca269c7164872bbfe60ef259f3a92, c844ad257bd04963a139ee84c3e43929; no voters:
I20260809 03:31:02.745909 1143 raft_consensus.cc:2781] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:02.746148 1143 raft_consensus.cc:686] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 1 LEADER]: Becoming Leader. State: Replica: c844ad257bd04963a139ee84c3e43929, State: Running, Role: LEADER
I20260809 03:31:02.746444 1143 consensus_queue.cc:237] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } }
I20260809 03:31:02.749611 419 catalog_manager.cc:5495] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 reported cstate change: term changed from 0 to 1, leader changed from <none> to c844ad257bd04963a139ee84c3e43929 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "c844ad257bd04963a139ee84c3e43929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c844ad257bd04963a139ee84c3e43929" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 44575 } health_report { overall_health: HEALTHY } } }
I20260809 03:31:02.827656 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.824592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.15.222.66:44575)
0809 03:31:02.824705 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:02.824708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:02.824723 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:31:02.824805 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:02.824807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:02.824862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:02.824950 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:02.825146 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.825626 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.826465 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:02.826673 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:02.826745 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:02.826968 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:02.827376 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:02.827459 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:02.827505 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:31:02.827551 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:31:02.833588 628 raft_consensus.cc:1261] T 20f7ca541a994ecc967c78b50092f215 P 3f4ca269c7164872bbfe60ef259f3a92 [term 1 FOLLOWER]: Refusing update from remote peer c844ad257bd04963a139ee84c3e43929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:02.833585 1043 raft_consensus.cc:1261] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 1 FOLLOWER]: Refusing update from remote peer c844ad257bd04963a139ee84c3e43929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:02.834329 1143 consensus_queue.cc:1035] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4ca269c7164872bbfe60ef259f3a92" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:02.834544 1150 consensus_queue.cc:1035] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417075d591f04f0c99c3f97ab44350fa" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:02.843379 1157 mvcc.cc:205] Tried to move back new op lower bound from 7316464692559036416 to 7316464692211912704. Current Snapshot: MvccSnapshot[applied={T|T < 7316464692559036416}]
I20260809 03:31:02.852097 418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:02.852268 418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:02.854701 418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb05c40437014e3bb73cf1ca76088c37: Sending DeleteTablet for 3 replicas of tablet 20f7ca541a994ecc967c78b50092f215
I20260809 03:31:02.855521 598 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f7ca541a994ecc967c78b50092f215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:02 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:49658
I20260809 03:31:02.855604 1009 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f7ca541a994ecc967c78b50092f215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:02 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:41286
I20260809 03:31:02.856010 800 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f7ca541a994ecc967c78b50092f215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:02 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:47470
I20260809 03:31:02.856237 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 506
I20260809 03:31:02.856789 1168 tablet_replica.cc:330] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: stopping tablet replica
I20260809 03:31:02.857105 1168 raft_consensus.cc:2229] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:02.857404 1168 raft_consensus.cc:2258] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:02.864534 1168 ts_tablet_manager.cc:1905] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:02.867334 1168 ts_tablet_manager.cc:1918] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:02.867442 1168 log.cc:1196] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/20f7ca541a994ecc967c78b50092f215
I20260809 03:31:02.867801 1168 ts_tablet_manager.cc:1939] T 20f7ca541a994ecc967c78b50092f215 P 417075d591f04f0c99c3f97ab44350fa: Deleting consensus metadata
I20260809 03:31:02.869093 394 catalog_manager.cc:4841] TS 417075d591f04f0c99c3f97ab44350fa (127.15.222.67:43333): tablet 20f7ca541a994ecc967c78b50092f215 (table test-table [id=ad531fefd61f4d4eb0f1f04ce3ddfc1f]) successfully deleted
W20260809 03:31:02.873431 394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:35539 (error 108)
I20260809 03:31:02.873692 1170 tablet_replica.cc:330] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929: stopping tablet replica
I20260809 03:31:02.874056 1170 raft_consensus.cc:2229] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:02.874451 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 707
I20260809 03:31:02.874500 1170 raft_consensus.cc:2258] T 20f7ca541a994ecc967c78b50092f215 P c844ad257bd04963a139ee84c3e43929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:02.874533 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.874236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:35539 (local address 127.0.0.1:33648)
0809 03:31:02.874358 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.874444 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 03:31:02.883279 394 catalog_manager.cc:4584] TS 3f4ca269c7164872bbfe60ef259f3a92 (127.15.222.65:35539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20f7ca541a994ecc967c78b50092f215: Network error: Client connection negotiation failed: client connection to 127.15.222.65:35539: connect: Connection refused (error 111)
I20260809 03:31:02.889698 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 913
I20260809 03:31:02.896755 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:02.889963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:44575 (local address 127.0.0.1:40188)
0809 03:31:02.896608 (+ 6645us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:02.896670 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.66:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6544}
I20260809 03:31:02.901324 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 362
2026-08-09T03:31:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2595 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[1179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[1179](info): set up 2 sockets
Aug 09 03:31:02 dist-test-slave-cvwc krb5kdc[1179](info): commencing operation
krb5kdc: starting...
W20260809 03:31:04.043697 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:31:04 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangeradmin_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangeradmin_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangerlookup_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangerlookup_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
I20260809 03:31:04.194671 16249 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbUfAuu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/postgres -l logfile start
2026-08-09 03:31:06.450 UTC [1224] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 03:31:06.450 UTC [1224] LOG: listening on IPv4 address "127.15.222.84", port 57505
2026-08-09 03:31:06.453 UTC [1224] LOG: listening on Unix socket "/tmp/.s.PGSQL.57505"
2026-08-09 03:31:06.481 UTC [1229] LOG: database system was shut down at 2026-08-09 03:31:05 UTC
2026-08-09 03:31:06.484 UTC [1224] LOG: database system is ready to accept connections
I20260809 03:31:07.537245 16249 mini_postgres.cc:96] Postgres bound to 57505
2026-08-09 03:31:07.545 UTC [1238] FATAL: database "slave" does not exist
127.15.222.84:57505 - accepting connections
I20260809 03:31:07.545926 16249 mini_ranger.cc:161] Starting Ranger...
I20260809 03:31:07.560858 16249 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 03:31:07.942682 16249 mini_ranger.cc:87] Created ranger Postgres database
I20260809 03:31:07.942767 16249 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin
2026-08-09 03:31:08,309 [I] DB FLAVOR :POSTGRES
2026-08-09 03:31:08,309 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:31:08,310 [I] Checking connection..
2026-08-09 03:31:08,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 03:31:08,548 [I] Checking connection passed.
2026-08-09 03:31:08,549 [I] --------- Verifying version history table ---------
2026-08-09 03:31:08,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:08,784 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:31:08,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:09,021 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:31:09,021 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 03:31:09,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 03:31:09.270 UTC [1344] WARNING: there is no transaction in progress
2026-08-09 03:31:09,278 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 03:31:09,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:09,535 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 03:31:09,536 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:31:09,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 03:31:09,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 03:31:10,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-09 03:31:10,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:31:10,505 [I] Table x_portal_user does not exist in database ranger
2026-08-09 03:31:10,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:31:10,738 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 03:31:10,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:31:10,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:31:11,229 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 03:31:11,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 03:31:12.093 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.108 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.122 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.135 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.149 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.168 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.225 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.230 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.243 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.252 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.260 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.272 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.281 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.286 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.292 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12.297 UTC [1508] WARNING: there is no transaction in progress
2026-08-09 03:31:12,842 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 03:31:12,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 03:31:13,072 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 03:31:13,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:31:13,303 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 03:31:13,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:31:13,566 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 03:31:13,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:31:13,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:31:14,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-09 03:31:14,296 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 03:31:14,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 03:31:14,531 [I] DB_PATCHES have already been applied
I20260809 03:31:14.536725 16249 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbUfAuu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 03:31:14.536791 16249 mini_ranger.cc:193] Using host: 127.15.222.84
I20260809 03:31:14.541006 16249 mini_ranger.cc:227] Ranger admin URL: http://127.15.222.84:44279
Aug 09, 2026 3:31:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:31:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:31:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:31:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:31:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:31:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:31:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 03:31:15.767686 16249 mini_ranger.cc:160] Time spent starting Ranger: real 8.222s user 0.000s sys 0.011s
Aug 09, 2026 3:31:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:31:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 3:31:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:31:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:31:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 03:31:37.299067 16249 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangerkms_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/rangerkms_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.222.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.222.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 03:31:37 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246297, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 03:31:37.381235 16249 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 03:31:37.391743 16249 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 03:31:38.102509 16249 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 03:31:38.102604 16249 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms
I20260809 03:31:38.102703 16249 mini_ranger_kms.cc:227] Using postgres at 127.15.222.84:57505
2026-08-09 03:31:39,206 [I] DB FLAVOR :POSTGRES
2026-08-09 03:31:39,206 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:31:39,206 [I] Checking connection
2026-08-09 03:31:39,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 03:31:39,558 [I] connection success
2026-08-09 03:31:39,558 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 03:31:39,558 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 03:31:39,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 03:31:39,832 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 03:31:39,832 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:31:39,832 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 03:31:39,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:57505/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 03:31:40,112 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 03:31:40.541819 16249 mini_ranger_kms.cc:310] Created kms service
I20260809 03:31:40.727784 16249 mini_ranger_kms.cc:326] Created kudu user
I20260809 03:31:40.770531 16249 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 03:31:40 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246297, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:41.159734 16249 mini_ranger_kms.cc:384] Added ranger policy
I20260809 03:31:41.159888 16249 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 03:31:41.159925 16249 mini_ranger_kms.cc:240] Using host: 127.15.222.84
I20260809 03:31:41.164989 16249 mini_ranger_kms.cc:276] Ranger KMS PID: 1774
I20260809 03:31:41.165097 16249 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.222.84:56917
Aug 09, 2026 3:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 03:31:43.616503 16249 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.235s user 0.014s sys 0.869s
I20260809 03:31:43.616639 16249 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 03:31:43.616758 16249 mini_ranger_kms.cc:401] 127.15.222.84:56917/kms/v1/keys
Aug 09 03:31:43 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246303, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 03:31:44 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246303, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
Aug 09 03:31:44 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246303, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
Aug 09 03:31:47 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:31:47.620199 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38227
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38227
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:56917/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:48.047855 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:48.048831 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:48.048998 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:48.063524 1864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:31:48.063791 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:48.063860 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:48.063990 1864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:31:48.064045 1864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:31:48.083681 1864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.222.84:56917/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38227
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:48.085632 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:48.087637 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:48.104104 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:48.104180 1875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:48.104213 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:48.104804 1864 server_base.cc:1034] running on GCE node
I20260809 03:31:48.105871 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:48.107798 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:48.109032 1864 hybrid_clock.cc:648] HybridClock initialized: now 1786246308108988 us; error 55 us; skew 500 ppm
Aug 09 03:31:48 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246308, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:48.117080 1864 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:31:48.118700 1864 webserver.cc:460] Webserver started at http://127.15.222.126:43839/ using document root <none> and password file <none>
I20260809 03:31:48.119617 1864 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:48.119720 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:48.120079 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:31:48 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246308, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:48.227681 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3636ec822289400ba5413dde0e44344e"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-cvwc"
server_key: "276ac4a48c0b70e1e73dbfb93dda017f"
server_key_iv: "739d19eba8cb45b360e8579d5d5f4825"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:48.229027 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3636ec822289400ba5413dde0e44344e"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-cvwc"
server_key: "276ac4a48c0b70e1e73dbfb93dda017f"
server_key_iv: "739d19eba8cb45b360e8579d5d5f4825"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:48.238955 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260809 03:31:48.274326 1887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:48.277859 1864 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260809 03:31:48.278095 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3636ec822289400ba5413dde0e44344e"
format_stamp: "Formatted at 2026-08-09 03:31:48 on dist-test-slave-cvwc"
server_key: "276ac4a48c0b70e1e73dbfb93dda017f"
server_key_iv: "739d19eba8cb45b360e8579d5d5f4825"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:48.278296 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:48.342391 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:48.348080 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:48.348426 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:48.375124 1864 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38227
I20260809 03:31:48.375312 1965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38227 every 8 connection(s)
I20260809 03:31:48.380995 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:31:48.388156 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 1864
I20260809 03:31:48.388623 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:31:48.389638 16249 external_mini_cluster.cc:1428] Setting key 0d40ee8ea6215acbcd17959317f02b55
I20260809 03:31:48.391637 1966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:48.405107 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: Bootstrap starting.
Aug 09 03:31:48 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:48.410079 1966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:48.413178 1966 log.cc:824] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:48.422919 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: No bootstrap required, opened a new log
I20260809 03:31:48.433795 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:48.393316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.15.222.126:38227)
0809 03:31:48.394399 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:48.394457 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:48.394547 (+ 90us) server_negotiation.cc:407] Connection header received
0809 03:31:48.396457 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:48.396512 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:48.397164 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:48.397891 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:48.399840 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.402074 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.403747 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:48.404705 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:48.405086 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:48.410196 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:48.410216 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:48.410247 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:48.410314 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:48.418272 (+ 7958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:48.420082 (+ 1810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:48.420137 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:48.420144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:48.420789 (+ 645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:48.421917 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:48.421921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:48.421923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:48.424599 (+ 2676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:48.425481 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:48.426584 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0809 03:31:48.428211 (+ 1627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":461,"threads_started":1}
I20260809 03:31:48.446566 1966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } }
I20260809 03:31:48.447111 1966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:48.447190 1966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3636ec822289400ba5413dde0e44344e, State: Initialized, Role: FOLLOWER
I20260809 03:31:48.447875 1966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } }
I20260809 03:31:48.448087 1966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:31:48.448165 1966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:31:48.448295 1966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:48.450378 1966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } }
I20260809 03:31:48.450970 1966 leader_election.cc:302] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3636ec822289400ba5413dde0e44344e; no voters:
I20260809 03:31:48.451471 1966 leader_election.cc:288] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:31:48.451745 1972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:48.452306 1972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [term 1 LEADER]: Becoming Leader. State: Replica: 3636ec822289400ba5413dde0e44344e, State: Running, Role: LEADER
I20260809 03:31:48.452883 1972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } }
I20260809 03:31:48.453788 1966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:31:48.457019 1975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3636ec822289400ba5413dde0e44344e. Latest consensus state: current_term: 1 leader_uuid: "3636ec822289400ba5413dde0e44344e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } } }
I20260809 03:31:48.457048 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3636ec822289400ba5413dde0e44344e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3636ec822289400ba5413dde0e44344e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38227 } } }
I20260809 03:31:48.457281 1975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:48.457281 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e [sys.catalog]: This master's current role is: LEADER
I20260809 03:31:48.459293 1986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:31:48.459887 1966 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:31:48.465854 1986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:31:48.496982 1986 catalog_manager.cc:1347] Generated new cluster ID: d4a4d2ae449c42858aafc01325c677e0
I20260809 03:31:48.497421 1986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:31:48.540935 1986 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:31:48.548085 1986 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:31:48.596612 1986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: Generated new TSK 0
I20260809 03:31:48.602313 1986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:31:49.621327 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:56917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:49.803048 2030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:49.803287 2030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:49.803328 2030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:49.808382 2030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:49.808466 2030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:49.808573 2030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:31:49.814054 2030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.222.84:56917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:49.815099 2030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:49.816136 2030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:49.829375 2041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:49.829854 2039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:49.830154 2030 server_base.cc:1034] running on GCE node
W20260809 03:31:49.831421 2038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:49.832038 2030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:49.832847 2030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:49.834009 2030 hybrid_clock.cc:648] HybridClock initialized: now 1786246309833978 us; error 49 us; skew 500 ppm
Aug 09 03:31:49 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246309, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:49.837627 2030 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:31:49.838734 2030 webserver.cc:460] Webserver started at http://127.15.222.65:37293/ using document root <none> and password file <none>
I20260809 03:31:49.839293 2030 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:49.839366 2030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:49.839568 2030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:31:49 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246309, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:49.856976 2030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "b599b08481484c0b86a981165c734c94"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-cvwc"
server_key: "617f6f9a95d0092adbee40f616420fef"
server_key_iv: "22286976928b4eddf4f44bc34c9cc3a1"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:49.857817 2030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "b599b08481484c0b86a981165c734c94"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-cvwc"
server_key: "617f6f9a95d0092adbee40f616420fef"
server_key_iv: "22286976928b4eddf4f44bc34c9cc3a1"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:49.867139 2030 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260809 03:31:49.888011 2059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:49.889830 2030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 03:31:49.889978 2030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "b599b08481484c0b86a981165c734c94"
format_stamp: "Formatted at 2026-08-09 03:31:49 on dist-test-slave-cvwc"
server_key: "617f6f9a95d0092adbee40f616420fef"
server_key_iv: "22286976928b4eddf4f44bc34c9cc3a1"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:49.890117 2030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:49.908360 2030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:49.911896 2030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:49.912137 2030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:49.912869 2030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:49.913975 2030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:49.914088 2030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:49.914213 2030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:49.914295 2030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:49.928843 2030 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:36199
I20260809 03:31:49.930174 2030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:31:49.934471 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 2030
I20260809 03:31:49.934598 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:31:49.934880 16249 external_mini_cluster.cc:1428] Setting key 4b5545b0bffa2300f1c46adc3c6825c5
I20260809 03:31:49.932278 2220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:36199 every 8 connection(s)
Aug 09 03:31:49 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246309, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:31:49.954133 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:49.934011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42865 (local address 127.15.222.126:38227)
0809 03:31:49.938693 (+ 4682us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:49.938698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:49.938712 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:31:49.938752 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:49.938755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:49.938802 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:49.938897 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:49.940391 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:49.941241 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:49.943182 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:49.943551 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:49.943645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:49.946754 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:49.946758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:49.946761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:49.946791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:49.948742 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:49.949344 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:49.949348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:49.949350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:49.949445 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:49.949954 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:49.949958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:49.949960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:49.952341 (+ 2381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:49.952444 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:49.953777 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0809 03:31:49.953934 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4600,"thread_start_us":82,"threads_started":1}
I20260809 03:31:49.955675 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:49.934525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38227 (local address 127.15.222.65:42865)
0809 03:31:49.937293 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:49.937334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:49.938262 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:49.939209 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:49.939218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:49.939569 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:49.940225 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:49.940244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:49.941388 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:49.941392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:49.942790 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:49.942802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:49.943813 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:49.943817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:49.944040 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:49.944717 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:49.944736 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:49.946552 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:49.948867 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:49.948873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:49.948908 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:49.949228 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:49.949553 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:49.949559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:49.949561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:49.949827 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:49.952504 (+ 2677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:49.952510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:49.952829 (+ 319us) client_negotiation.cc:769] Sending connection context
0809 03:31:49.953065 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 03:31:49.953389 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2586,"thread_start_us":137,"threads_started":1}
I20260809 03:31:49.957445 2221 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38227
I20260809 03:31:49.957700 2221 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:49.958307 2221 heartbeater.cc:511] Master 127.15.222.126:38227 requested a full tablet report, sending...
I20260809 03:31:49.960024 1913 ts_manager.cc:194] Registered new tserver with Master: b599b08481484c0b86a981165c734c94 (127.15.222.65:36199)
I20260809 03:31:49.961565 1913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:42865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:31:49.992190 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:56917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 03:31:50.157233 2232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:50.157485 2232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:50.157544 2232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:50.161586 2232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:50.161656 2232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:50.161752 2232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:31:50.166983 2232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.222.84:56917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:50.168035 2232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:50.169230 2232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:50.179273 2244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.179286 2241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.179718 2242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:50.180084 2232 server_base.cc:1034] running on GCE node
I20260809 03:31:50.180478 2232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:50.181178 2232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:31:50.182363 2232 hybrid_clock.cc:648] HybridClock initialized: now 1786246310182336 us; error 36 us; skew 500 ppm
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:50.185290 2232 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:31:50.186424 2232 webserver.cc:460] Webserver started at http://127.15.222.66:35887/ using document root <none> and password file <none>
I20260809 03:31:50.187049 2232 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:50.187119 2232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:50.187492 2232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:50.203550 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "3f8e8591f4fb451fb8dfea094452019e"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "5af49c2abe88f4cf4fb62bf1d4e1f742"
server_key_iv: "23dfcd88955bf82ac515030a8f2a1b6b"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.204465 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "3f8e8591f4fb451fb8dfea094452019e"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "5af49c2abe88f4cf4fb62bf1d4e1f742"
server_key_iv: "23dfcd88955bf82ac515030a8f2a1b6b"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.211428 2232 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20260809 03:31:50.234766 2254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.236451 2232 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260809 03:31:50.236598 2232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "3f8e8591f4fb451fb8dfea094452019e"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "5af49c2abe88f4cf4fb62bf1d4e1f742"
server_key_iv: "23dfcd88955bf82ac515030a8f2a1b6b"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.236761 2232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:50.315112 2232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:50.318888 2232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:50.319196 2232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:50.319959 2232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:50.321112 2232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:50.321182 2232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.321246 2232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:50.321282 2232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.333361 2232 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42243
I20260809 03:31:50.334607 2232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:31:50.343039 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 2232
I20260809 03:31:50.343155 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:31:50.343474 16249 external_mini_cluster.cc:1428] Setting key 70deb60094a2dee5659c01dbfecbdd68
I20260809 03:31:50.354503 2431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42243 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:50.383195 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.352809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:57919 (local address 127.15.222.126:38227)
0809 03:31:50.352937 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.352941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.358186 (+ 5245us) server_negotiation.cc:407] Connection header received
0809 03:31:50.365371 (+ 7185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.365375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.365438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.365536 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:50.367223 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.368040 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.369811 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.370180 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.370275 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.373253 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:50.373257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:50.373260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:50.373288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:50.375563 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.381120 (+ 5557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.381125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.381127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.381236 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.381774 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.381777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.381779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.382023 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:50.382126 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.382847 (+ 721us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.383001 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:31:50.384227 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.353403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38227 (local address 127.15.222.66:57919)
0809 03:31:50.358015 (+ 4612us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.358075 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.365101 (+ 7026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.365919 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.365930 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.366312 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:50.367035 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.367057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.368203 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.368206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.369643 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.369655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.370494 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.370498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.370694 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.371369 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:50.371389 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:50.373048 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:50.375704 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:50.375710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:50.375727 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:50.376090 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:50.381366 (+ 5276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:50.381369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:50.381371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:50.381585 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:50.382156 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:50.382164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:50.382531 (+ 367us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.382753 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.383126 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4373,"thread_start_us":149,"threads_started":1}
I20260809 03:31:50.385489 2432 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38227
I20260809 03:31:50.385771 2432 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:50.386389 2432 heartbeater.cc:511] Master 127.15.222.126:38227 requested a full tablet report, sending...
I20260809 03:31:50.387590 1913 ts_manager.cc:194] Registered new tserver with Master: 3f8e8591f4fb451fb8dfea094452019e (127.15.222.66:42243)
I20260809 03:31:50.388361 1913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:57919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:31:50.393402 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:56917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:31:50.533051 2438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:31:50.533319 2438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:31:50.533389 2438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:31:50.538290 2438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:31:50.538359 2438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:31:50.538443 2438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:31:50.544538 2438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.222.84:56917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:31:50.545658 2438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:31:50.546856 2438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:31:50.557967 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.558598 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:31:50.558871 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:31:50.560007 2438 server_base.cc:1034] running on GCE node
I20260809 03:31:50.560683 2438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:31:50.561502 2438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:50.577184 2438 hybrid_clock.cc:648] HybridClock initialized: now 1786246310577131 us; error 66 us; skew 500 ppm
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:31:50.580931 2438 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:31:50.582159 2438 webserver.cc:460] Webserver started at http://127.15.222.67:46679/ using document root <none> and password file <none>
I20260809 03:31:50.582983 2438 fs_manager.cc:359] Metadata directory not provided
I20260809 03:31:50.583142 2438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:31:50.583467 2438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:50.611009 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "5f988fb69d6243589814f8859b31b10b"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "1f5e94ff5933e20fb0acdd72a67eeaba"
server_key_iv: "9091ca474a277cc46a656978f94a6463"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.611744 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "5f988fb69d6243589814f8859b31b10b"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "1f5e94ff5933e20fb0acdd72a67eeaba"
server_key_iv: "9091ca474a277cc46a656978f94a6463"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.624006 2438 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.004s
I20260809 03:31:50.646096 2456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.647765 2438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 03:31:50.647905 2438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "5f988fb69d6243589814f8859b31b10b"
format_stamp: "Formatted at 2026-08-09 03:31:50 on dist-test-slave-cvwc"
server_key: "1f5e94ff5933e20fb0acdd72a67eeaba"
server_key_iv: "9091ca474a277cc46a656978f94a6463"
server_key_version: "kuduclusterkey@0"
I20260809 03:31:50.648059 2438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:31:50.663241 2438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:31:50.666702 2438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:31:50.666898 2438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:31:50.667623 2438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:31:50.668650 2438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:31:50.668712 2438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.668776 2438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:31:50.668804 2438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:31:50.682502 2438 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:42223
I20260809 03:31:50.682649 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:42223 every 8 connection(s)
I20260809 03:31:50.683656 2438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:31:50.690666 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 2438
I20260809 03:31:50.690789 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:31:50.691062 16249 external_mini_cluster.cc:1428] Setting key 3574bed57319c8259a86f7588c54c090
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:31:50.699751 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.687215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:57963 (local address 127.15.222.126:38227)
0809 03:31:50.687418 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.687423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.687441 (+ 18us) server_negotiation.cc:407] Connection header received
0809 03:31:50.688154 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.688157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.688210 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.688315 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:50.689639 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.690543 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.692229 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.692607 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.692680 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.695687 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:50.695691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:50.695694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:50.695723 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:50.697377 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.698007 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.698011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.698013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.698099 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.698500 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.698503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.698505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.698719 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:50.698811 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.699469 (+ 658us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.699593 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260809 03:31:50.700994 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.686204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38227 (local address 127.15.222.67:57963)
0809 03:31:50.686818 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.686884 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.687927 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.688494 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.688503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.688846 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:31:50.689480 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.689499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.690699 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.690703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.692082 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.692095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.692879 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.692883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.693090 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.693757 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:31:50.693778 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:31:50.695491 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:31:50.697502 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:50.697508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:50.697523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:50.697908 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:50.698195 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:31:50.698198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:31:50.698200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:31:50.698388 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:31:50.698836 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:31:50.698843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:31:50.699164 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.699388 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.699730 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":132,"threads_started":1}
I20260809 03:31:50.702142 2613 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38227
I20260809 03:31:50.702379 2613 heartbeater.cc:464] Registering TS with master...
I20260809 03:31:50.702877 2613 heartbeater.cc:511] Master 127.15.222.126:38227 requested a full tablet report, sending...
I20260809 03:31:50.703965 1913 ts_manager.cc:194] Registered new tserver with Master: 5f988fb69d6243589814f8859b31b10b (127.15.222.67:42223)
I20260809 03:31:50.704646 1913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:57963
I20260809 03:31:50.705019 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:31:50.715178 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.707417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.15.222.126:38227)
0809 03:31:50.707517 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.707521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.707698 (+ 177us) server_negotiation.cc:407] Connection header received
0809 03:31:50.707801 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.707804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.707844 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.707917 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:31:50.708525 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.709291 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.710142 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.710486 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.710591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.711907 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:31:50.711911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:31:50.711913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:31:50.711938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:31:50.713452 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.713923 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.713927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.713929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.714024 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:31:50.714263 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:31:50.714266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:31:50.714268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:31:50.714489 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:31:50.714629 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.714898 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.715021 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:31:50.718264 1913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:31:50.722512 1913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:31:50.736843 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.730552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:36199 (local address 127.0.0.1:55162)
0809 03:31:50.731203 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.731251 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.731388 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.732438 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.732445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.732469 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:50.732600 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.732606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.733742 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.733745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.735056 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.735066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.736327 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.736331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.736434 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.736474 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.736574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.736691 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":99,"threads_started":1}
I20260809 03:31:50.737471 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.731678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.15.222.65:36199)
0809 03:31:50.731941 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.731947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.731971 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:31:50.732050 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.732056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.732257 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.732748 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:50.732868 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.733600 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.735210 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.736015 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.736110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.736262 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.737232 (+ 970us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.737335 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260809 03:31:50.740514 2123 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d (DEFAULT_TABLE table=test-table [id=c64ab6f7e8214f21acd10e49715b6bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:50.741483 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.731413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42243 (local address 127.0.0.1:42422)
0809 03:31:50.734991 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.735004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.735118 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.735790 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.735793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.735810 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:50.735891 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.735896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.738706 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.738709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.739875 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.739886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.741110 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.741114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.741197 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.741219 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.741294 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.741362 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3520,"mutex_wait_us":33,"thread_start_us":90,"threads_started":1}
I20260809 03:31:50.742045 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.732117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.15.222.66:42243)
0809 03:31:50.732739 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.732754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.735210 (+ 2456us) server_negotiation.cc:407] Connection header received
0809 03:31:50.735300 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.735305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.735483 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.735654 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:50.736015 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.738353 (+ 2338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.740036 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.740781 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.740878 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.741005 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.741818 (+ 813us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.741912 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":83,"threads_started":1}
I20260809 03:31:50.742187 2123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:50.744827 2323 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d (DEFAULT_TABLE table=test-table [id=c64ab6f7e8214f21acd10e49715b6bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:50.745966 2323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:50.748857 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.730691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:42223 (local address 127.0.0.1:33730)
0809 03:31:50.732447 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.732459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.732527 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.738978 (+ 6451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.738981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.738999 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:50.739150 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.739158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.740526 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.740529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.741645 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.741657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.748469 (+ 6812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.748473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.748572 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.748594 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.748657 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.748724 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260809 03:31:50.749344 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.731174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.15.222.67:42223)
0809 03:31:50.738337 (+ 7163us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.738376 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.738407 (+ 31us) server_negotiation.cc:407] Connection header received
0809 03:31:50.738495 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.738500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.738668 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.738835 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:50.739282 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.740383 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.747378 (+ 6995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.748154 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.748257 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.748387 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.749112 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.749212 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7061,"thread_start_us":221,"threads_started":1}
I20260809 03:31:50.754737 2639 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Bootstrap starting.
I20260809 03:31:50.755066 2522 tablet_service.cc:1467] Processing CreateTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d (DEFAULT_TABLE table=test-table [id=c64ab6f7e8214f21acd10e49715b6bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:31:50.756338 2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:31:50.756454 2638 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Bootstrap starting.
I20260809 03:31:50.757519 2639 tablet_bootstrap.cc:654] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:50.759042 2639 log.cc:824] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:50.759538 2638 tablet_bootstrap.cc:654] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:50.760360 2638 log.cc:824] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:50.761327 2639 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: No bootstrap required, opened a new log
I20260809 03:31:50.761504 2639 ts_tablet_manager.cc:1397] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 03:31:50.762351 2638 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: No bootstrap required, opened a new log
I20260809 03:31:50.762552 2638 ts_tablet_manager.cc:1397] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:31:50.766201 2639 raft_consensus.cc:348] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.766458 2639 raft_consensus.cc:374] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:50.766515 2639 raft_consensus.cc:729] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8e8591f4fb451fb8dfea094452019e, State: Initialized, Role: FOLLOWER
I20260809 03:31:50.766981 2638 raft_consensus.cc:348] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.767225 2638 raft_consensus.cc:374] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:50.767275 2638 raft_consensus.cc:729] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b599b08481484c0b86a981165c734c94, State: Initialized, Role: FOLLOWER
I20260809 03:31:50.767359 2639 consensus_queue.cc:260] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.768606 2639 ts_tablet_manager.cc:1428] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 03:31:50.768651 2638 consensus_queue.cc:260] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.769071 2432 heartbeater.cc:503] Master 127.15.222.126:38227 was elected leader, sending a full tablet report...
I20260809 03:31:50.769582 2638 ts_tablet_manager.cc:1428] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 03:31:50.770138 2221 heartbeater.cc:503] Master 127.15.222.126:38227 was elected leader, sending a full tablet report...
I20260809 03:31:50.774124 2644 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Bootstrap starting.
I20260809 03:31:50.777089 2644 tablet_bootstrap.cc:654] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Neither blocks nor log segments found. Creating new log.
I20260809 03:31:50.777855 2644 log.cc:824] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:31:50.779628 2644 tablet_bootstrap.cc:492] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: No bootstrap required, opened a new log
I20260809 03:31:50.779814 2644 ts_tablet_manager.cc:1397] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260809 03:31:50.782766 2644 raft_consensus.cc:348] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.783036 2644 raft_consensus.cc:374] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:31:50.783088 2644 raft_consensus.cc:729] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f988fb69d6243589814f8859b31b10b, State: Initialized, Role: FOLLOWER
I20260809 03:31:50.783592 2644 consensus_queue.cc:260] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.784297 2613 heartbeater.cc:503] Master 127.15.222.126:38227 was elected leader, sending a full tablet report...
I20260809 03:31:50.784551 2644 ts_tablet_manager.cc:1428] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
W20260809 03:31:50.851531 2433 tablet.cc:2369] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:50.857079 2651 raft_consensus.cc:482] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:31:50.857304 2651 raft_consensus.cc:504] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.858733 2651 leader_election.cc:288] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b599b08481484c0b86a981165c734c94 (127.15.222.65:36199), 3f8e8591f4fb451fb8dfea094452019e (127.15.222.66:42243)
I20260809 03:31:50.864076 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.859027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:36199 (local address 127.15.222.67:49161)
0809 03:31:50.859343 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.859360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.859465 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.860272 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.860275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.860295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:50.860411 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.860418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.861398 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.861402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.862620 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.862632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.863651 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.863655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.863756 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.863779 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.863847 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.863924 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260809 03:31:50.864558 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.859672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:49161 (local address 127.15.222.65:36199)
0809 03:31:50.859855 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.859860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.859875 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:31:50.859926 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.859929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.859977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.860048 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:50.860543 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.861276 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.862792 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.863404 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.863533 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.863588 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.864349 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.864426 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 03:31:50.865677 2131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11e9827e5c4635aa08cf4e1ed80b2d" candidate_uuid: "5f988fb69d6243589814f8859b31b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b599b08481484c0b86a981165c734c94" is_pre_election: true
I20260809 03:31:50.866061 2131 raft_consensus.cc:2454] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f988fb69d6243589814f8859b31b10b in term 0.
I20260809 03:31:50.866609 2457 leader_election.cc:302] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f988fb69d6243589814f8859b31b10b, b599b08481484c0b86a981165c734c94; no voters:
I20260809 03:31:50.866930 2651 raft_consensus.cc:2781] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:31:50.867020 2651 raft_consensus.cc:482] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:31:50.867066 2651 raft_consensus.cc:3037] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:50.868398 2651 raft_consensus.cc:504] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.868999 2131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11e9827e5c4635aa08cf4e1ed80b2d" candidate_uuid: "5f988fb69d6243589814f8859b31b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b599b08481484c0b86a981165c734c94"
I20260809 03:31:50.869135 2131 raft_consensus.cc:3037] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:50.870334 2131 raft_consensus.cc:2454] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f988fb69d6243589814f8859b31b10b in term 1.
I20260809 03:31:50.870654 2457 leader_election.cc:302] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f988fb69d6243589814f8859b31b10b, b599b08481484c0b86a981165c734c94; no voters:
I20260809 03:31:50.870999 2651 leader_election.cc:288] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [CANDIDATE]: Term 1 election: Requested vote from peers b599b08481484c0b86a981165c734c94 (127.15.222.65:36199), 3f8e8591f4fb451fb8dfea094452019e (127.15.222.66:42243)
I20260809 03:31:50.871088 2651 raft_consensus.cc:2781] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:31:50.871153 2651 raft_consensus.cc:686] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 1 LEADER]: Becoming Leader. State: Replica: 5f988fb69d6243589814f8859b31b10b, State: Running, Role: LEADER
I20260809 03:31:50.871501 2651 consensus_queue.cc:237] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } }
I20260809 03:31:50.874778 1909 catalog_manager.cc:5495] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f988fb69d6243589814f8859b31b10b (127.15.222.67). New cstate: current_term: 1 leader_uuid: "5f988fb69d6243589814f8859b31b10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f988fb69d6243589814f8859b31b10b" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 42223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 } health_report { overall_health: UNKNOWN } } }
I20260809 03:31:50.877622 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.859718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:34465 (local address 127.15.222.66:42243)
0809 03:31:50.860218 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.860223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.863384 (+ 3161us) server_negotiation.cc:407] Connection header received
0809 03:31:50.863579 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.863582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.863636 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.863686 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:31:50.864154 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.864855 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.876267 (+ 11412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.876867 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.876981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.877037 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.877414 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.877500 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
I20260809 03:31:50.878214 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.859305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42243 (local address 127.15.222.67:34465)
0809 03:31:50.863289 (+ 3984us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:50.863304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:31:50.863459 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:31:50.863835 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:31:50.863838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:31:50.863858 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:31:50.864011 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.864018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.871259 (+ 7241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.871262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.876105 (+ 4843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:31:50.876118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.877162 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.877165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:31:50.877262 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.877285 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:31:50.877973 (+ 688us) client_negotiation.cc:240] Negotiation successful
0809 03:31:50.878056 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3909,"thread_start_us":131,"threads_started":1}
I20260809 03:31:50.878856 2360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11e9827e5c4635aa08cf4e1ed80b2d" candidate_uuid: "5f988fb69d6243589814f8859b31b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8e8591f4fb451fb8dfea094452019e" is_pre_election: true
I20260809 03:31:50.878878 2359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c11e9827e5c4635aa08cf4e1ed80b2d" candidate_uuid: "5f988fb69d6243589814f8859b31b10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8e8591f4fb451fb8dfea094452019e"
I20260809 03:31:50.879133 2360 raft_consensus.cc:2454] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f988fb69d6243589814f8859b31b10b in term 0.
Aug 09 03:31:50 dist-test-slave-cvwc krb5kdc[1179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246310, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:31:50.908110 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:50.902994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.15.222.67:42223)
0809 03:31:50.903138 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 03:31:50.903142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:31:50.903156 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:31:50.903239 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:31:50.903243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:31:50.903309 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:31:50.903411 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:31:50.903815 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.904559 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.905915 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:31:50.906264 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:31:50.906361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:31:50.906817 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:31:50.907381 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:31:50.907505 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 03:31:50.907909 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 03:31:50.907970 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 03:31:50.919085 2359 raft_consensus.cc:3037] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:31:50.919219 2131 raft_consensus.cc:1261] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 1 FOLLOWER]: Refusing update from remote peer 5f988fb69d6243589814f8859b31b10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:50.920629 2359 raft_consensus.cc:1261] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 1 FOLLOWER]: Refusing update from remote peer 5f988fb69d6243589814f8859b31b10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:31:50.921527 2653 consensus_queue.cc:1035] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8e8591f4fb451fb8dfea094452019e" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:50.921692 2651 consensus_queue.cc:1035] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b599b08481484c0b86a981165c734c94" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:31:50.931670 2659 mvcc.cc:205] Tried to move back new op lower bound from 7316464889510776832 to 7316464889332948992. Current Snapshot: MvccSnapshot[applied={T|T < 7316464889510776832}]
I20260809 03:31:50.934351 2661 mvcc.cc:205] Tried to move back new op lower bound from 7316464889510776832 to 7316464889332948992. Current Snapshot: MvccSnapshot[applied={T|T < 7316464889510776832}]
W20260809 03:31:50.942493 2222 tablet.cc:2369] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:31:50.945587 2662 mvcc.cc:205] Tried to move back new op lower bound from 7316464889510776832 to 7316464889332948992. Current Snapshot: MvccSnapshot[applied={T|T < 7316464889510776832}]
I20260809 03:31:50.949112 1913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:50.949287 1913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:31:50.951910 1913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3636ec822289400ba5413dde0e44344e: Sending DeleteTablet for 3 replicas of tablet 2c11e9827e5c4635aa08cf4e1ed80b2d
I20260809 03:31:50.953023 2522 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:50 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:33730
I20260809 03:31:50.953526 2323 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:50 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42422
I20260809 03:31:50.954316 2123 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:31:50 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:55162
I20260809 03:31:50.955000 2676 tablet_replica.cc:330] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94: stopping tablet replica
I20260809 03:31:50.955332 2676 raft_consensus.cc:2229] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:50.955471 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 2030
I20260809 03:31:50.955649 2676 raft_consensus.cc:2258] T 2c11e9827e5c4635aa08cf4e1ed80b2d P b599b08481484c0b86a981165c734c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:50.970764 2670 tablet_replica.cc:330] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: stopping tablet replica
I20260809 03:31:50.971177 2670 raft_consensus.cc:2229] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:31:50.971587 2670 raft_consensus.cc:2258] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:50.972453 2671 tablet_replica.cc:330] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: stopping tablet replica
I20260809 03:31:50.972761 2671 raft_consensus.cc:2229] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:31:50.974201 2671 raft_consensus.cc:2258] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:31:50.975986 2671 ts_tablet_manager.cc:1905] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:50.979506 2671 ts_tablet_manager.cc:1918] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:50.979604 2671 log.cc:1196] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/2c11e9827e5c4635aa08cf4e1ed80b2d
I20260809 03:31:50.979885 2671 ts_tablet_manager.cc:1939] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 3f8e8591f4fb451fb8dfea094452019e: Deleting consensus metadata
I20260809 03:31:50.981263 1891 catalog_manager.cc:4841] TS 3f8e8591f4fb451fb8dfea094452019e (127.15.222.66:42243): tablet 2c11e9827e5c4635aa08cf4e1ed80b2d (table test-table [id=c64ab6f7e8214f21acd10e49715b6bb6]) successfully deleted
W20260809 03:31:50.981755 1889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:36199 (error 108)
I20260809 03:31:50.982359 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 2232
I20260809 03:31:50.996258 2670 ts_tablet_manager.cc:1905] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:31:50.999831 2670 ts_tablet_manager.cc:1918] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:31:51.000025 2670 log.cc:1196] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/2c11e9827e5c4635aa08cf4e1ed80b2d
I20260809 03:31:51.000396 2670 ts_tablet_manager.cc:1939] T 2c11e9827e5c4635aa08cf4e1ed80b2d P 5f988fb69d6243589814f8859b31b10b: Deleting consensus metadata
I20260809 03:31:51.001705 1892 catalog_manager.cc:4841] TS 5f988fb69d6243589814f8859b31b10b (127.15.222.67:42223): tablet 2c11e9827e5c4635aa08cf4e1ed80b2d (table test-table [id=c64ab6f7e8214f21acd10e49715b6bb6]) successfully deleted
I20260809 03:31:51.002934 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:31:51.002554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:36199 (local address 127.0.0.1:55168)
0809 03:31:51.002716 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 03:31:51.002827 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 03:31:51.003249 1889 catalog_manager.cc:4584] TS b599b08481484c0b86a981165c734c94 (127.15.222.65:36199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c11e9827e5c4635aa08cf4e1ed80b2d: Network error: Client connection negotiation failed: client connection to 127.15.222.65:36199: connect: Connection refused (error 111)
I20260809 03:31:51.033741 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 2438
I20260809 03:31:51.050609 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 1864
I20260809 03:31:51.065953 16249 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 03:31:51.412684 16249 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 03:31:51.412811 16249 mini_ranger.cc:66] Stopping Ranger...
I20260809 03:31:51.756811 16249 mini_ranger.cc:68] Stopped Ranger
2026-08-09 03:31:51.757 UTC [1224] LOG: received smart shutdown request
2026-08-09 03:31:51.760 UTC [1224] LOG: background worker "logical replication launcher" (PID 1236) exited with exit code 1
2026-08-09 03:31:51.761 UTC [1231] LOG: shutting down
2026-08-09 03:31:51.918 UTC [1224] LOG: database system is shut down
2026-08-09T03:31:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49115 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:31:52 dist-test-slave-cvwc krb5kdc[2692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:31:52 dist-test-slave-cvwc krb5kdc[2692](info): set up 2 sockets
Aug 09 03:31:52 dist-test-slave-cvwc krb5kdc[2692](info): commencing operation
krb5kdc: starting...
W20260809 03:31:53.181432 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:31:53 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:31:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangeradmin_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangeradmin_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangerlookup_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangerlookup_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
I20260809 03:31:53.346707 16249 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbUfAuu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/postgres -l logfile start
2026-08-09 03:31:55.610 UTC [2741] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 03:31:55.610 UTC [2741] LOG: listening on IPv4 address "127.15.222.84", port 36279
2026-08-09 03:31:55.613 UTC [2741] LOG: listening on Unix socket "/tmp/.s.PGSQL.36279"
2026-08-09 03:31:55.633 UTC [2744] LOG: database system was shut down at 2026-08-09 03:31:54 UTC
2026-08-09 03:31:55.636 UTC [2741] LOG: database system is ready to accept connections
I20260809 03:31:56.688349 16249 mini_postgres.cc:96] Postgres bound to 36279
2026-08-09 03:31:56.695 UTC [2753] FATAL: database "slave" does not exist
127.15.222.84:36279 - accepting connections
I20260809 03:31:56.695886 16249 mini_ranger.cc:161] Starting Ranger...
I20260809 03:31:56.710350 16249 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 03:31:57.098376 16249 mini_ranger.cc:87] Created ranger Postgres database
I20260809 03:31:57.098464 16249 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin
2026-08-09 03:31:57,380 [I] DB FLAVOR :POSTGRES
2026-08-09 03:31:57,380 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:31:57,380 [I] Checking connection..
2026-08-09 03:31:57,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 03:31:57,637 [I] Checking connection passed.
2026-08-09 03:31:57,637 [I] --------- Verifying version history table ---------
2026-08-09 03:31:57,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:57,888 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:31:57,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:58,144 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 03:31:58,144 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 03:31:58,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 03:31:58.393 UTC [2859] WARNING: there is no transaction in progress
2026-08-09 03:31:58,400 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 03:31:58,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 03:31:58,661 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 03:31:58,661 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:31:58,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 03:31:58,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 03:31:59,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-09 03:31:59,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:31:59,653 [I] Table x_portal_user does not exist in database ranger
2026-08-09 03:31:59,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:31:59,914 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 03:31:59,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:32:00,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:32:00,378 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 03:32:00,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 03:32:01.233 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.248 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.262 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.276 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.289 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.307 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.361 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.368 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.381 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.390 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.399 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.411 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.420 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.426 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.431 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01.437 UTC [3006] WARNING: there is no transaction in progress
2026-08-09 03:32:01,975 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 03:32:01,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 03:32:02,223 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 03:32:02,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 03:32:02,468 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 03:32:02,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 03:32:02,710 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 03:32:02,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 03:32:02,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 03:32:03,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-09 03:32:03,405 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 03:32:03,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 03:32:03,679 [I] DB_PATCHES have already been applied
I20260809 03:32:03.685037 16249 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbUfAuu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 03:32:03.685103 16249 mini_ranger.cc:193] Using host: 127.15.222.84
I20260809 03:32:03.689792 16249 mini_ranger.cc:227] Ranger admin URL: http://127.15.222.84:38407
Aug 09, 2026 3:32:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:32:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:32:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 3:32:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 3:32:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 3:32:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 3:32:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 03:32:04.890379 16249 mini_ranger.cc:160] Time spent starting Ranger: real 8.194s user 0.003s sys 0.009s
Aug 09, 2026 3:32:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 3:32:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 3:32:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 3:32:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 3:32:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 03:32:24.569901 16249 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangerkms_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/rangerkms_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.222.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.222.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.222.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/HTTP_127.15.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 03:32:24 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 03:32:24.649163 16249 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 03:32:24.658970 16249 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 03:32:25.343524 16249 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 03:32:25.343606 16249 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms
I20260809 03:32:25.343621 16249 mini_ranger_kms.cc:227] Using postgres at 127.15.222.84:36279
2026-08-09 03:32:25,511 [I] DB FLAVOR :POSTGRES
2026-08-09 03:32:25,512 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 03:32:25,512 [I] Checking connection
2026-08-09 03:32:25,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 03:32:25,760 [I] connection success
2026-08-09 03:32:25,761 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 03:32:25,761 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 03:32:25,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 03:32:26,002 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 03:32:26,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 03:32:26,002 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 03:32:26,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.222.84:36279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 03:32:26,252 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 03:32:26.703929 16249 mini_ranger_kms.cc:310] Created kms service
I20260809 03:32:26.867777 16249 mini_ranger_kms.cc:326] Created kudu user
I20260809 03:32:26.910759 16249 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 03:32:26 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:27.267489 16249 mini_ranger_kms.cc:384] Added ranger policy
I20260809 03:32:27.267745 16249 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbUfAuu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbUfAuu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 03:32:27.267781 16249 mini_ranger_kms.cc:240] Using host: 127.15.222.84
I20260809 03:32:27.272389 16249 mini_ranger_kms.cc:276] Ranger KMS PID: 3267
I20260809 03:32:27.272471 16249 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.222.84:51629
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 3:32:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 03:32:29 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:29.733871 16249 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.085s user 0.004s sys 0.150s
I20260809 03:32:29.734020 16249 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 03:32:29.734107 16249 mini_ranger_kms.cc:401] 127.15.222.84:51629/kms/v1/keys
Aug 09 03:32:29 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
Aug 09 03:32:30 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.222.84@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
Aug 09 03:32:32 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:32:32.412361 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44299
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:44299
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:32.549357 3336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:32.549608 3336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:32.549646 3336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:32.553565 3336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:32.553624 3336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:32.553642 3336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:32.553658 3336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:32.553671 3336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:32.559038 3336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.222.84:51629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:44299
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:32.560041 3336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:32.561043 3336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:32.569653 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:32.569697 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:32.569844 3336 server_base.cc:1034] running on GCE node
W20260809 03:32:32.569662 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:32.570643 3336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:32.572002 3336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:32.573203 3336 hybrid_clock.cc:648] HybridClock initialized: now 1786246352573173 us; error 38 us; skew 500 ppm
Aug 09 03:32:32 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246352, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:32.581065 3336 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:32.582353 3336 webserver.cc:460] Webserver started at http://127.15.222.126:37679/ using document root <none> and password file <none>
I20260809 03:32:32.583043 3336 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:32.583128 3336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:32.583390 3336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:32:32 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246352, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:32.632463 3336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "ffac365c698c4d338154e42fd75f01dd"
format_stamp: "Formatted at 2026-08-09 03:32:32 on dist-test-slave-cvwc"
server_key: "c52e5fa0644677a5ac64735bfab39f93"
server_key_iv: "0ef1f5d480ee5915d9e0087bae193246"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:32.633332 3336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "ffac365c698c4d338154e42fd75f01dd"
format_stamp: "Formatted at 2026-08-09 03:32:32 on dist-test-slave-cvwc"
server_key: "c52e5fa0644677a5ac64735bfab39f93"
server_key_iv: "0ef1f5d480ee5915d9e0087bae193246"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:32.638481 3336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 03:32:32.652371 3360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:32.653690 3336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:32.653844 3336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "ffac365c698c4d338154e42fd75f01dd"
format_stamp: "Formatted at 2026-08-09 03:32:32 on dist-test-slave-cvwc"
server_key: "c52e5fa0644677a5ac64735bfab39f93"
server_key_iv: "0ef1f5d480ee5915d9e0087bae193246"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:32.653976 3336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:32.665845 3336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:32.668599 3336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:32.668758 3336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:32.676302 3336 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:44299
I20260809 03:32:32.676308 3426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:44299 every 8 connection(s)
I20260809 03:32:32.677251 3336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:32.680112 3428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:32.680426 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 3336
I20260809 03:32:32.680544 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:32:32.680869 16249 external_mini_cluster.cc:1428] Setting key ef04758a4e6c5d8f864e5971d099b5b9
I20260809 03:32:32.687198 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Bootstrap starting.
I20260809 03:32:32.689801 3428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:32.690757 3428 log.cc:824] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:32:32 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:32.692852 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: No bootstrap required, opened a new log
I20260809 03:32:32.696224 3428 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:32.696451 3428 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:32.696523 3428 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffac365c698c4d338154e42fd75f01dd, State: Initialized, Role: FOLLOWER
I20260809 03:32:32.696969 3428 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:32.697110 3428 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:32.697188 3428 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:32.697292 3428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:32.698194 3428 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:32.698555 3428 leader_election.cc:302] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffac365c698c4d338154e42fd75f01dd; no voters:
I20260809 03:32:32.698868 3428 leader_election.cc:288] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:32.698967 3437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:32.698993 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:32.682377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.15.222.126:44299)
0809 03:32:32.682855 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:32.682874 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:32.682916 (+ 42us) server_negotiation.cc:407] Connection header received
0809 03:32:32.683818 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:32.683843 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:32.684275 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:32.684649 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:32.685529 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:32.687087 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:32.688261 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:32.688956 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:32.689110 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:32.692040 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:32.692047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:32.692065 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:32.692097 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:32.694848 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:32.695363 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:32.695370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:32.695377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:32.695526 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:32.695833 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:32.695837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:32.695839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:32.696392 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:32.697464 (+ 1072us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:32.697643 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 03:32:32.697997 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260809 03:32:32.699213 3437 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 LEADER]: Becoming Leader. State: Replica: ffac365c698c4d338154e42fd75f01dd, State: Running, Role: LEADER
I20260809 03:32:32.700627 3437 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:32.700766 3428 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:32.701978 3437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffac365c698c4d338154e42fd75f01dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } } }
I20260809 03:32:32.702162 3437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:32.702683 3439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffac365c698c4d338154e42fd75f01dd. Latest consensus state: current_term: 1 leader_uuid: "ffac365c698c4d338154e42fd75f01dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } } }
I20260809 03:32:32.702836 3439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:32.702970 3428 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:32:32.703088 3452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:32.710999 3452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:32.723316 3452 catalog_manager.cc:1347] Generated new cluster ID: d5425a02e90c45d1a307c7c6027ed871
I20260809 03:32:32.723395 3452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:32.746412 3452 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:32.747788 3452 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:32.757287 3452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Generated new TSK 0
I20260809 03:32:32.758425 3452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:32:33.468776 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:33.585325 3496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:33.585569 3496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:33.585610 3496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:33.589548 3496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:33.589607 3496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:33.589684 3496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:32:33.595031 3496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:33.596024 3496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:33.597077 3496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:33.607007 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:33.607035 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:33.607041 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:33.607573 3496 server_base.cc:1034] running on GCE node
I20260809 03:32:33.607924 3496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:33.608511 3496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:33.609675 3496 hybrid_clock.cc:648] HybridClock initialized: now 1786246353609661 us; error 24 us; skew 500 ppm
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:33.612385 3496 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:32:33.613175 3496 webserver.cc:460] Webserver started at http://127.15.222.65:46819/ using document root <none> and password file <none>
I20260809 03:32:33.613799 3496 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:33.613894 3496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:33.614157 3496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:33.632319 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "34ea29ae21d34ad29d9a5fce95677fd4"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "f58a449f165635498e422aa2a3245ac6"
server_key_iv: "19d5f9ac4fcb288c9b2c9df54696a97c"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:33.633080 3496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "34ea29ae21d34ad29d9a5fce95677fd4"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "f58a449f165635498e422aa2a3245ac6"
server_key_iv: "19d5f9ac4fcb288c9b2c9df54696a97c"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:33.638258 3496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 03:32:33.658099 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:33.659847 3496 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260809 03:32:33.659983 3496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "34ea29ae21d34ad29d9a5fce95677fd4"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "f58a449f165635498e422aa2a3245ac6"
server_key_iv: "19d5f9ac4fcb288c9b2c9df54696a97c"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:33.660135 3496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:33.694069 3496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:33.697662 3496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:33.697957 3496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:33.698798 3496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:33.699973 3496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:33.700105 3496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:33.700248 3496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:33.700350 3496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:33.712746 3496 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:33429
I20260809 03:32:33.714047 3496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:32:33.717314 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 3496
I20260809 03:32:33.717458 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:32:33.717816 16249 external_mini_cluster.cc:1428] Setting key dfa06eb53c7c1f63a4680088890e70ec
I20260809 03:32:33.724197 3685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:33429 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:33.751422 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:33.726841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.65:54221)
0809 03:32:33.734833 (+ 7992us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:33.734900 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:33.736028 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:33.736823 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:33.736832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:33.737162 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:33.737890 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:33.737912 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:33.738986 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:33.738990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:33.740348 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:33.740361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:33.741132 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:33.741136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:33.741335 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:33.742111 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:33.742134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:33.745740 (+ 3606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:33.747917 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:33.747923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:33.747941 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:33.748284 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:33.748732 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:33.748736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:33.748738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:33.748926 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:33.749467 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:33.749474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:33.749846 (+ 372us) client_negotiation.cc:769] Sending connection context
0809 03:32:33.750046 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 03:32:33.750383 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7736,"thread_start_us":146,"threads_started":1}
I20260809 03:32:33.752346 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:33.725932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:54221 (local address 127.15.222.126:44299)
0809 03:32:33.729938 (+ 4006us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:33.729947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:33.735045 (+ 5098us) server_negotiation.cc:407] Connection header received
0809 03:32:33.736316 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:33.736320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:33.736379 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:33.736488 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:33.738071 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:33.738822 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:33.740492 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:33.740847 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:33.740950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:33.745960 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:33.745963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:33.745966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:33.745995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:33.747779 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:33.748415 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:33.748418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:33.748420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:33.748566 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:33.749039 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:33.749043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:33.749045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:33.749345 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:33.749455 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:33.752045 (+ 2590us) server_negotiation.cc:299] Negotiation successful
0809 03:32:33.752178 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3913,"thread_start_us":115,"threads_started":1}
I20260809 03:32:33.753505 3686 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:33.753779 3686 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:33.754365 3686 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:33.756222 3378 ts_manager.cc:194] Registered new tserver with Master: 34ea29ae21d34ad29d9a5fce95677fd4 (127.15.222.65:33429)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:32:33.758415 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:33.760403 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:54221
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 03:32:33.916787 3694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:33.917122 3694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:33.917187 3694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:33.923154 3694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:33.923274 3694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:33.923381 3694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:32:33.930045 3694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:33.931077 3694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:33.932070 3694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:33.947255 3705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:33.947288 3702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:33.948133 3703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:33.948916 3694 server_base.cc:1034] running on GCE node
I20260809 03:32:33.949487 3694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:33.950274 3694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:33.951439 3694 hybrid_clock.cc:648] HybridClock initialized: now 1786246353951375 us; error 79 us; skew 500 ppm
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:33.954922 3694 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:32:33.956040 3694 webserver.cc:460] Webserver started at http://127.15.222.66:44293/ using document root <none> and password file <none>
I20260809 03:32:33.956846 3694 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:33.956936 3694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:33.957267 3694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:32:33 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:33.984818 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "c42f019b277b493eb607730d7ba652bf"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "cd58e31dc1de512f46165652cc8212f0"
server_key_iv: "dcbddbf104e3bef38c2161845a2e6d38"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:33.985677 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "c42f019b277b493eb607730d7ba652bf"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "cd58e31dc1de512f46165652cc8212f0"
server_key_iv: "dcbddbf104e3bef38c2161845a2e6d38"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:33.991175 3694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 03:32:34.022461 3716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.025498 3694 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260809 03:32:34.025666 3694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "c42f019b277b493eb607730d7ba652bf"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "cd58e31dc1de512f46165652cc8212f0"
server_key_iv: "dcbddbf104e3bef38c2161845a2e6d38"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:34.025856 3694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:34.044656 3694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:34.048285 3694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:34.048476 3694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:34.049152 3694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:34.050230 3694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:34.050303 3694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.050359 3694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:34.050386 3694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.060737 3694 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:37829
I20260809 03:32:34.061961 3875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:37829 every 8 connection(s)
I20260809 03:32:34.062016 3694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:32:34.062538 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 3694
I20260809 03:32:34.062660 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:32:34.062966 16249 external_mini_cluster.cc:1428] Setting key e772c937ebf47b056c3c7c78e6a838da
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:32:34.081423 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:34.067256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:59771 (local address 127.15.222.126:44299)
0809 03:32:34.067400 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:34.067403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:34.067693 (+ 290us) server_negotiation.cc:407] Connection header received
0809 03:32:34.068873 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:34.068876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:34.068938 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:34.069031 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:34.070938 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.071761 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.073467 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.073855 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.073940 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:34.077033 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:34.077037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:34.077040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:34.077069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:34.078794 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:34.079448 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:34.079452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:34.079454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:34.079547 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:34.079977 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:34.079980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:34.079982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:34.080213 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:34.080287 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:34.081064 (+ 777us) server_negotiation.cc:299] Negotiation successful
0809 03:32:34.081201 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
I20260809 03:32:34.082293 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:34.065697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.66:59771)
0809 03:32:34.067498 (+ 1801us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:34.067568 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:34.068619 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:34.069411 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:34.069422 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:34.069875 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:34.070742 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:34.070767 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.071928 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.071932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:34.073303 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:34.073315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.074151 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.074154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:34.074362 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:34.075063 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:34.075083 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:34.076842 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:34.078941 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:34.078947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:34.078964 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:34.079315 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:34.079664 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:34.079667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:34.079669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:34.079859 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:34.080372 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:34.080378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:34.080708 (+ 330us) client_negotiation.cc:769] Sending connection context
0809 03:32:34.080923 (+ 215us) client_negotiation.cc:240] Negotiation successful
0809 03:32:34.081256 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":142,"threads_started":1}
I20260809 03:32:34.085222 3877 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:34.085471 3877 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:34.086072 3877 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:34.087242 3378 ts_manager.cc:194] Registered new tserver with Master: c42f019b277b493eb607730d7ba652bf (127.15.222.66:37829)
I20260809 03:32:34.087988 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:59771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:32:34.109505 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246353, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
W20260809 03:32:34.263465 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:34.263943 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:34.264075 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:34.269763 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:34.269877 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:34.269991 3883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:32:34.277336 3883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:34.278858 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:34.280229 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:34.293193 3893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.293612 3889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.294123 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:34.294119 3883 server_base.cc:1034] running on GCE node
I20260809 03:32:34.294636 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:34.295351 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:34.300218 3883 hybrid_clock.cc:648] HybridClock initialized: now 1786246354300132 us; error 94 us; skew 500 ppm
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:34.305748 3883 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:32:34.306845 3883 webserver.cc:460] Webserver started at http://127.15.222.67:40673/ using document root <none> and password file <none>
I20260809 03:32:34.307554 3883 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:34.307638 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:34.307859 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:34.354238 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "e6a239474b5f40ed9b748a1fe53c7b7a"
format_stamp: "Formatted at 2026-08-09 03:32:34 on dist-test-slave-cvwc"
server_key: "7596e525f707e1ae3fb5a11e71c43fb5"
server_key_iv: "fdffbb876cb5f3e2111c23a5f01b9eb7"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:34.354969 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "e6a239474b5f40ed9b748a1fe53c7b7a"
format_stamp: "Formatted at 2026-08-09 03:32:34 on dist-test-slave-cvwc"
server_key: "7596e525f707e1ae3fb5a11e71c43fb5"
server_key_iv: "fdffbb876cb5f3e2111c23a5f01b9eb7"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:34.365476 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260809 03:32:34.400677 3903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.402684 3883 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260809 03:32:34.402837 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "e6a239474b5f40ed9b748a1fe53c7b7a"
format_stamp: "Formatted at 2026-08-09 03:32:34 on dist-test-slave-cvwc"
server_key: "7596e525f707e1ae3fb5a11e71c43fb5"
server_key_iv: "fdffbb876cb5f3e2111c23a5f01b9eb7"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:34.402989 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:34.439244 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:34.442741 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:34.442937 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:34.443612 3883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:34.444669 3883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:34.444727 3883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.444788 3883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:34.444816 3883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.455526 3883 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:33753
I20260809 03:32:34.455665 4049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:33753 every 8 connection(s)
I20260809 03:32:34.456626 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:32:34.464540 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 3883
I20260809 03:32:34.464669 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:32:34.464999 16249 external_mini_cluster.cc:1428] Setting key 5fbccf0fdd2dcb84159f8b345bee159f
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:34.472334 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:34.458556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:52493 (local address 127.15.222.126:44299)
0809 03:32:34.458683 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:34.458687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:34.459638 (+ 951us) server_negotiation.cc:407] Connection header received
0809 03:32:34.460913 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:34.460916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:34.460971 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:34.461022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:34.462621 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.463415 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.465027 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.465365 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.465439 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:34.468523 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:34.468527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:34.468530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:34.468558 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:34.470101 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:34.470659 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:34.470663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:34.470665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:34.470745 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:34.471123 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:34.471126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:34.471128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:34.471335 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:34.471404 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:34.472067 (+ 663us) server_negotiation.cc:299] Negotiation successful
0809 03:32:34.472183 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:32:34.474114 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:34.458943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.67:52493)
0809 03:32:34.459479 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:34.459536 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:34.460633 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:34.461429 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:34.461437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:34.461822 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:34.462459 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:34.462475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.463565 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.463569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:34.464893 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:34.464904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:34.465763 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:34.465780 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:34.465982 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:34.466648 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:34.466675 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:34.468335 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:34.470232 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:34.470238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:34.470253 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:34.470556 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:34.470861 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:34.470863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:34.470866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:34.471028 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:34.471484 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:34.471490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:34.471787 (+ 297us) client_negotiation.cc:769] Sending connection context
0809 03:32:34.472737 (+ 950us) client_negotiation.cc:240] Negotiation successful
0809 03:32:34.473065 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260809 03:32:34.475728 4050 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:34.475994 4050 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:34.476577 4050 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:34.477978 3378 ts_manager.cc:194] Registered new tserver with Master: e6a239474b5f40ed9b748a1fe53c7b7a (127.15.222.67:33753)
I20260809 03:32:34.478731 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:52493
I20260809 03:32:34.487471 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:32:34.488459 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 3496
I20260809 03:32:34.531670 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 3694
I20260809 03:32:34.545327 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 3883
I20260809 03:32:34.566337 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 3336
I20260809 03:32:34.580969 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44299
--webserver_interface=127.15.222.126
--webserver_port=37679
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:44299
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:34.695900 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:34.696185 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:34.696233 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:34.700284 4055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:34.700348 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:34.700371 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:34.700388 4055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:34.700407 4055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:34.705878 4055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:44299
--ranger_config_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=37679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:34.706883 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:34.707926 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:34.716076 4070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.716086 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.716077 4068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:34.716455 4055 server_base.cc:1034] running on GCE node
I20260809 03:32:34.716985 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:34.718060 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:34.719215 4055 hybrid_clock.cc:648] HybridClock initialized: now 1786246354719203 us; error 34 us; skew 500 ppm
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:34.721817 4055 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:34.722585 4055 webserver.cc:460] Webserver started at http://127.15.222.126:37679/ using document root <none> and password file <none>
I20260809 03:32:34.723044 4055 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:34.723105 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:34.760859 4055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 03:32:34.763046 4082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:34.764262 4055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:32:34.764369 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "ffac365c698c4d338154e42fd75f01dd"
format_stamp: "Formatted at 2026-08-09 03:32:32 on dist-test-slave-cvwc"
server_key: "c52e5fa0644677a5ac64735bfab39f93"
server_key_iv: "0ef1f5d480ee5915d9e0087bae193246"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:34.764901 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:34.779263 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:34.782002 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:34.782151 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:34.788797 4055 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:44299
I20260809 03:32:34.788806 4166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:44299 every 8 connection(s)
I20260809 03:32:34.789885 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:34.794019 4167 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:32:34.795549 4167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Bootstrap starting.
I20260809 03:32:34.798432 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 4055
I20260809 03:32:34.798835 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:33429
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:44299
--webserver_port=46819
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:34.811735 4167 log.cc:824] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:34.818130 4167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:34.818485 4167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Bootstrap complete.
I20260809 03:32:34.822325 4167 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:34.822710 4167 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffac365c698c4d338154e42fd75f01dd, State: Initialized, Role: FOLLOWER
I20260809 03:32:34.823177 4167 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:34.823300 4167 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:34.823366 4167 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:34.823467 4167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:32:34.825219 4167 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:34.825524 4167 leader_election.cc:302] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffac365c698c4d338154e42fd75f01dd; no voters:
I20260809 03:32:34.825845 4167 leader_election.cc:288] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:32:34.825937 4171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:32:34.826109 4171 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [term 2 LEADER]: Becoming Leader. State: Replica: ffac365c698c4d338154e42fd75f01dd, State: Running, Role: LEADER
I20260809 03:32:34.826390 4171 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } }
I20260809 03:32:34.827018 4167 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:34.827656 4172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffac365c698c4d338154e42fd75f01dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } } }
I20260809 03:32:34.827776 4172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:34.828361 4173 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffac365c698c4d338154e42fd75f01dd. Latest consensus state: current_term: 2 leader_uuid: "ffac365c698c4d338154e42fd75f01dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffac365c698c4d338154e42fd75f01dd" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 44299 } } }
I20260809 03:32:34.828503 4173 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:34.828856 4179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:34.829180 4167 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 03:32:34.834901 4179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:34.835602 4179 catalog_manager.cc:1259] Loaded cluster ID: d5425a02e90c45d1a307c7c6027ed871
I20260809 03:32:34.835692 4179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:34.838207 4179 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:34.838927 4179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Loaded TSK: 0
I20260809 03:32:34.840166 4179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:32:34.928582 4169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:34.928882 4169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:34.928939 4169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:34.934855 4169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:34.934917 4169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:34.934998 4169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:32:34.942454 4169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:33429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=46819
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:34.943512 4169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:34.944706 4169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:34.954025 4225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.954025 4222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:34.954380 4221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:34.954954 4169 server_base.cc:1034] running on GCE node
I20260809 03:32:34.955353 4169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:34.955992 4169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:34.957170 4169 hybrid_clock.cc:648] HybridClock initialized: now 1786246354957136 us; error 47 us; skew 500 ppm
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:34.959980 4169 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:32:34.960867 4169 webserver.cc:460] Webserver started at http://127.15.222.65:46819/ using document root <none> and password file <none>
I20260809 03:32:34.961468 4169 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:34.961576 4169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:32:34 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:35.004537 4169 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.003s sys 0.000s
I20260809 03:32:35.007704 4237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.009218 4169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 03:32:35.009351 4169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "34ea29ae21d34ad29d9a5fce95677fd4"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "f58a449f165635498e422aa2a3245ac6"
server_key_iv: "19d5f9ac4fcb288c9b2c9df54696a97c"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:35.009981 4169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:35.051386 4169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:35.054970 4169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:35.055172 4169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:35.055917 4169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:35.056982 4169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:35.057050 4169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.057113 4169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:35.057144 4169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.068588 4169 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:33429
I20260809 03:32:35.068743 4406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:33429 every 8 connection(s)
I20260809 03:32:35.070075 4169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:32:35.076661 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 4169
I20260809 03:32:35.077145 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:37829
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:44299
--webserver_port=44293
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:35.093341 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.072920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:39267 (local address 127.15.222.126:44299)
0809 03:32:35.075218 (+ 2298us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:35.075236 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:35.075269 (+ 33us) server_negotiation.cc:407] Connection header received
0809 03:32:35.075498 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:35.075508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:35.075907 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:35.076265 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:35.077363 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.079465 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.081135 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.081896 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.082032 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.087104 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:35.087111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:35.087127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:35.087160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:35.089238 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.089925 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.089932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.089938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.090087 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.090527 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.090531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.090533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.091025 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:35.091149 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:35.091924 (+ 775us) server_negotiation.cc:299] Negotiation successful
0809 03:32:35.092251 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2159,"thread_start_us":118,"threads_started":1}
I20260809 03:32:35.095115 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.65:39267)
0809 03:32:35.074031 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:35.074073 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:35.074824 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:35.076347 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:35.076355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:35.076682 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:35.077172 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.077186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.079620 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.079625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.080985 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.080998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.082281 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.082284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.082482 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.083094 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:35.083114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:35.086879 (+ 3765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:35.089378 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.089396 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.089411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.089781 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.090212 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.090215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.090217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.090411 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.091222 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:35.091228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:35.091547 (+ 319us) client_negotiation.cc:769] Sending connection context
0809 03:32:35.093871 (+ 2324us) client_negotiation.cc:240] Negotiation successful
0809 03:32:35.094196 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":125,"threads_started":1}
I20260809 03:32:35.098042 4409 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:35.098301 4409 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.098891 4409 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
W20260809 03:32:35.101259 4409 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:44299 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 03:32:35.101383 4409 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.101557 4409 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:35.102501 4409 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.102679 4409 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
W20260809 03:32:35.103442 4409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:32:35.222909 4415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:35.223136 4415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:35.223176 4415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:35.227608 4415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:35.227666 4415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:35.227742 4415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:32:35.234478 4415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:37829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=44293
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:35.235715 4415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:35.237061 4415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:35.248116 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:35.248232 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:35.248484 4415 server_base.cc:1034] running on GCE node
W20260809 03:32:35.249832 4423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:35.250237 4415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:35.250847 4415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:35.252024 4415 hybrid_clock.cc:648] HybridClock initialized: now 1786246355252002 us; error 32 us; skew 500 ppm
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:35.254782 4415 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:32:35.255788 4415 webserver.cc:460] Webserver started at http://127.15.222.66:44293/ using document root <none> and password file <none>
I20260809 03:32:35.256469 4415 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:35.256562 4415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:35.282881 4415 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260809 03:32:35.287350 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.288668 4415 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 03:32:35.288786 4415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "c42f019b277b493eb607730d7ba652bf"
format_stamp: "Formatted at 2026-08-09 03:32:33 on dist-test-slave-cvwc"
server_key: "cd58e31dc1de512f46165652cc8212f0"
server_key_iv: "dcbddbf104e3bef38c2161845a2e6d38"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:35.289345 4415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:35.314476 4415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:35.317939 4415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:35.318147 4415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:35.318857 4415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:35.319869 4415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:35.319947 4415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.320005 4415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:35.320036 4415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.331128 4415 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:37829
I20260809 03:32:35.331172 4601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:37829 every 8 connection(s)
I20260809 03:32:35.332139 4415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:32:35.333809 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 4415
I20260809 03:32:35.334267 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.222.84:51629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:33753
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:44299
--webserver_port=40673
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:35.351876 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.335619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:55671 (local address 127.15.222.126:44299)
0809 03:32:35.335766 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:35.335770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:35.336129 (+ 359us) server_negotiation.cc:407] Connection header received
0809 03:32:35.337334 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:35.337337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:35.337395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:35.337447 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:35.338881 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.339404 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.340739 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.341056 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.341149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.345549 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:35.345553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:35.345556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:35.345585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:35.347258 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.350039 (+ 2781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.350043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.350045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.350135 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.350540 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.350544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.350546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.350773 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:35.350862 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:35.351563 (+ 701us) server_negotiation.cc:299] Negotiation successful
0809 03:32:35.351709 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:32:35.357856 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.334573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.66:55671)
0809 03:32:35.335962 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:35.336023 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:35.337091 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:35.337912 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:35.337921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:35.338239 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:35.338738 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.338751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.339522 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.339526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.340622 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.340632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.341197 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.341202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.341390 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.342007 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:35.342027 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:35.345352 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:35.347402 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.347409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.347424 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.349918 (+ 2494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.350240 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.350243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.350245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.350428 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.350936 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:35.350942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:35.351253 (+ 311us) client_negotiation.cc:769] Sending connection context
0809 03:32:35.353277 (+ 2024us) client_negotiation.cc:240] Negotiation successful
0809 03:32:35.353616 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":139,"threads_started":1}
I20260809 03:32:35.359174 4602 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:35.359465 4602 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.360041 4602 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
W20260809 03:32:35.361943 4602 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:44299 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 03:32:35.362072 4602 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.362241 4602 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:35.363114 4602 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.363281 4602 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
W20260809 03:32:35.364078 4602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:32:35.472568 4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:35.472882 4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:35.472944 4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:35.478953 4605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:35.479031 4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:35.479112 4605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:32:35.486434 4605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.222.84:51629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:33753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=40673
--tserver_master_addrs=127.15.222.126:44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:35.487505 4605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:35.488531 4605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:35.498679 4613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:35.498668 4617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:35.499409 4611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:35.499657 4605 server_base.cc:1034] running on GCE node
I20260809 03:32:35.500108 4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:35.500818 4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:35.504092 4605 hybrid_clock.cc:648] HybridClock initialized: now 1786246355504054 us; error 37 us; skew 500 ppm
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:35.507746 4605 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:32:35.508554 4605 webserver.cc:460] Webserver started at http://127.15.222.67:40673/ using document root <none> and password file <none>
I20260809 03:32:35.509078 4605 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:35.509148 4605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:35.534611 4605 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260809 03:32:35.538645 4637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.540231 4605 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 03:32:35.540365 4605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "e6a239474b5f40ed9b748a1fe53c7b7a"
format_stamp: "Formatted at 2026-08-09 03:32:34 on dist-test-slave-cvwc"
server_key: "7596e525f707e1ae3fb5a11e71c43fb5"
server_key_iv: "fdffbb876cb5f3e2111c23a5f01b9eb7"
server_key_version: "kuduclusterkey@0"
I20260809 03:32:35.541100 4605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:35.581908 4605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:35.585193 4605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:35.585381 4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:35.586052 4605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:35.586928 4605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:35.586977 4605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.587031 4605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:35.587059 4605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:35.597124 4605 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:33753
I20260809 03:32:35.597165 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:33753 every 8 connection(s)
I20260809 03:32:35.598191 4605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:32:35.599182 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 4605
I20260809 03:32:35.609313 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.599874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.15.222.126:44299)
0809 03:32:35.600020 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:35.600025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:35.600721 (+ 696us) server_negotiation.cc:407] Connection header received
0809 03:32:35.600782 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:35.600785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:35.600840 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:35.600921 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:35.601938 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.602717 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.603941 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.604284 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.604409 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.605968 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:35.605972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:35.605975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:35.606001 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:35.607552 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.608020 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.608023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.608025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.608125 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.608390 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.608394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.608395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.608641 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:35.608841 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:35.609036 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 03:32:35.609148 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":27520}
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:35.614454 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.601070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:53261 (local address 127.15.222.126:44299)
0809 03:32:35.602959 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:35.602966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:35.602978 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:35.604682 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:35.604685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:35.604726 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:35.604795 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:35.606231 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.606887 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.608138 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.608447 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.608531 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.610977 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:35.610981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:35.610983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:35.611009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:35.612433 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.612893 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.612896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.612898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.612985 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:35.613346 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:35.613349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:35.613351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:35.613563 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:35.613664 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:35.614156 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 03:32:35.614288 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":94,"threads_started":1}
I20260809 03:32:35.615127 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:35.601479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:44299 (local address 127.15.222.67:53261)
0809 03:32:35.602203 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:35.602260 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:35.604475 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:35.605129 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:35.605139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:35.605465 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:35.606099 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.606115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.607008 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.607012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.608035 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:35.608046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:35.608586 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:35.608590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:35.608784 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:35.609334 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:35.609356 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:35.610834 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:35.612537 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.612541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.612552 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.612786 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.613060 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:35.613062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:35.613063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:35.613232 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:35.613655 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:35.613660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:35.613917 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 03:32:35.614060 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 03:32:35.614290 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":162,"threads_started":1}
I20260809 03:32:35.616133 4832 heartbeater.cc:347] Connected to a master server at 127.15.222.126:44299
I20260809 03:32:35.616372 4832 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:35.616922 4832 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:35.618336 4112 ts_manager.cc:194] Registered new tserver with Master: e6a239474b5f40ed9b748a1fe53c7b7a (127.15.222.67:33753)
I20260809 03:32:35.619910 4112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:53261
Aug 09 03:32:35 dist-test-slave-cvwc krb5kdc[2692](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:36.103672 4409 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:36.103996 4409 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:36.104727 4112 ts_manager.cc:194] Registered new tserver with Master: 34ea29ae21d34ad29d9a5fce95677fd4 (127.15.222.65:33429)
I20260809 03:32:36.105540 4112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:39267
Aug 09 03:32:36 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
I20260809 03:32:36.364277 4602 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:36.364555 4602 heartbeater.cc:511] Master 127.15.222.126:44299 requested a full tablet report, sending...
I20260809 03:32:36.365268 4110 ts_manager.cc:194] Registered new tserver with Master: c42f019b277b493eb607730d7ba652bf (127.15.222.66:37829)
I20260809 03:32:36.365622 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:32:36.366042 4110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:55671
I20260809 03:32:36.375988 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.367856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.15.222.126:44299)
0809 03:32:36.368075 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.368079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.368092 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:32:36.368127 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.368130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.368169 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.369099 (+ 930us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:36.369196 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.370025 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.371085 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.371302 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.371369 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.372868 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:36.372872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:36.372874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:36.372895 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:36.374495 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:36.374939 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:36.374941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:36.374943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:36.375011 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:36.375272 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:36.375274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:36.375275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:36.375423 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:36.375517 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.375745 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.375850 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260809 03:32:36.379485 4110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:32:36.383021 4110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:32:36.400483 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.392984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:37829 (local address 127.0.0.1:58014)
0809 03:32:36.393601 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.393649 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:36.393914 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:36.395600 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:36.395608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:36.395635 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:36.395774 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.395780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.397419 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.397423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.398813 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.398824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.399942 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.399946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.400053 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.400115 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 03:32:36.400223 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 03:32:36.400347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":103,"threads_started":1}
I20260809 03:32:36.401666 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.393107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.15.222.66:37829)
0809 03:32:36.395072 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.395079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.395105 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:32:36.395185 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.395191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.395415 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.395955 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:36.396093 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.397278 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.398978 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.399804 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.399925 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.400077 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.401437 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.401536 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1857,"thread_start_us":104,"threads_started":1}
I20260809 03:32:36.404709 4496 tablet_service.cc:1467] Processing CreateTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f (DEFAULT_TABLE table=test-table [id=b9136f38d8554c5d8040fc29826a8209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:36.405848 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f529cec21a4cc9b21a846d4b31bc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:36.409337 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.395270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33429 (local address 127.0.0.1:58932)
0809 03:32:36.400521 (+ 5251us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.400533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:36.400607 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:36.401398 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:36.401400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:36.401417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:36.401505 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.401510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.403788 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.403792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.404967 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.404979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.408961 (+ 3982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.408964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.409055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.409079 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:32:36.409145 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 03:32:36.409208 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5197,"thread_start_us":95,"threads_started":1}
I20260809 03:32:36.411193 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.394092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:33753 (local address 127.0.0.1:36488)
0809 03:32:36.400256 (+ 6164us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.400271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:36.400356 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:36.402349 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:36.402352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:36.402370 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:36.402452 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.402458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.406549 (+ 4091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.406552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.407674 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.407686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.410798 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.410802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.410904 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.410927 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:32:36.411002 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:32:36.411069 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6092,"thread_start_us":88,"threads_started":1}
I20260809 03:32:36.411598 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.395685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.15.222.65:33429)
0809 03:32:36.400828 (+ 5143us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.400835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.400857 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:32:36.400938 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.400943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.401127 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.401260 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:36.402584 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.403653 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.407817 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.408637 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.408741 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.408892 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.411367 (+ 2475us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.411464 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5040,"thread_start_us":102,"threads_started":1}
I20260809 03:32:36.412027 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.394540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.15.222.67:33753)
0809 03:32:36.401788 (+ 7248us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.401794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.401819 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:32:36.401900 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.401904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.402082 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.402212 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:36.405119 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.406414 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.409606 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.410515 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.410615 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.410737 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.411782 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.411880 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7108,"thread_start_us":121,"threads_started":1}
I20260809 03:32:36.414418 4288 tablet_service.cc:1467] Processing CreateTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f (DEFAULT_TABLE table=test-table [id=b9136f38d8554c5d8040fc29826a8209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:36.414795 4732 tablet_service.cc:1467] Processing CreateTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f (DEFAULT_TABLE table=test-table [id=b9136f38d8554c5d8040fc29826a8209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:36.415462 4288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f529cec21a4cc9b21a846d4b31bc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:36.415762 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f529cec21a4cc9b21a846d4b31bc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:36.425697 4870 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Bootstrap starting.
I20260809 03:32:36.427951 4866 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Bootstrap starting.
I20260809 03:32:36.428339 4870 tablet_bootstrap.cc:654] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:36.429138 4870 log.cc:824] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:36.430069 4866 tablet_bootstrap.cc:654] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:36.430879 4866 log.cc:824] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:36.431564 4871 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Bootstrap starting.
I20260809 03:32:36.431596 4870 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: No bootstrap required, opened a new log
I20260809 03:32:36.431777 4870 ts_tablet_manager.cc:1397] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 03:32:36.433662 4871 tablet_bootstrap.cc:654] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:36.434463 4871 log.cc:824] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:36.435401 4870 raft_consensus.cc:348] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.435652 4870 raft_consensus.cc:374] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:36.435705 4870 raft_consensus.cc:729] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ea29ae21d34ad29d9a5fce95677fd4, State: Initialized, Role: FOLLOWER
I20260809 03:32:36.436528 4866 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: No bootstrap required, opened a new log
I20260809 03:32:36.436717 4866 ts_tablet_manager.cc:1397] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260809 03:32:36.436896 4870 consensus_queue.cc:260] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.437923 4870 ts_tablet_manager.cc:1428] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260809 03:32:36.438253 4409 heartbeater.cc:503] Master 127.15.222.126:44299 was elected leader, sending a full tablet report...
I20260809 03:32:36.441213 4866 raft_consensus.cc:348] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.441488 4866 raft_consensus.cc:374] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:36.441557 4866 raft_consensus.cc:729] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a239474b5f40ed9b748a1fe53c7b7a, State: Initialized, Role: FOLLOWER
I20260809 03:32:36.442076 4866 consensus_queue.cc:260] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.443071 4866 ts_tablet_manager.cc:1428] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 03:32:36.443429 4832 heartbeater.cc:503] Master 127.15.222.126:44299 was elected leader, sending a full tablet report...
I20260809 03:32:36.444366 4871 tablet_bootstrap.cc:492] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: No bootstrap required, opened a new log
I20260809 03:32:36.444566 4871 ts_tablet_manager.cc:1397] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20260809 03:32:36.448052 4871 raft_consensus.cc:348] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.448244 4871 raft_consensus.cc:374] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:36.448309 4871 raft_consensus.cc:729] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f019b277b493eb607730d7ba652bf, State: Initialized, Role: FOLLOWER
I20260809 03:32:36.448670 4871 consensus_queue.cc:260] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.449236 4602 heartbeater.cc:503] Master 127.15.222.126:44299 was elected leader, sending a full tablet report...
I20260809 03:32:36.449496 4871 ts_tablet_manager.cc:1428] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
Aug 09 03:32:36 dist-test-slave-cvwc krb5kdc[2692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for HTTP/127.15.222.84@KRBTEST.COM
W20260809 03:32:36.573465 4410 tablet.cc:2369] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:32:36.584290 4603 tablet.cc:2369] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:32:36.600471 4833 tablet.cc:2369] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:36.660239 4878 raft_consensus.cc:482] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:36.660491 4878 raft_consensus.cc:504] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.662026 4878 leader_election.cc:288] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6a239474b5f40ed9b748a1fe53c7b7a (127.15.222.67:33753), 34ea29ae21d34ad29d9a5fce95677fd4 (127.15.222.65:33429)
I20260809 03:32:36.672516 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.662776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:52451 (local address 127.15.222.67:33753)
0809 03:32:36.663023 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.663027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.664884 (+ 1857us) server_negotiation.cc:407] Connection header received
0809 03:32:36.665116 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.665119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.665168 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.665217 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:36.665746 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.666556 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.668316 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.668940 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.669038 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.669094 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.672293 (+ 3199us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.672380 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260809 03:32:36.672986 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.662254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:33753 (local address 127.15.222.66:52451)
0809 03:32:36.664771 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.664786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:36.664980 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:36.665418 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:36.665422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:36.665443 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:36.665589 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.665596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.666731 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.666735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.668098 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.668110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.672044 (+ 3934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.672047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.672143 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.672169 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 03:32:36.672732 (+ 563us) client_negotiation.cc:240] Negotiation successful
0809 03:32:36.672807 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2440,"thread_start_us":147,"threads_started":1}
I20260809 03:32:36.673290 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.662772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:56789 (local address 127.15.222.65:33429)
0809 03:32:36.662932 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.662937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.668251 (+ 5314us) server_negotiation.cc:407] Connection header received
0809 03:32:36.669216 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.669220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.669274 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.669351 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:36.669763 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.670509 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.672019 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.672628 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.672744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.672793 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.673095 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.673172 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 03:32:36.673664 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.662372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33429 (local address 127.15.222.66:56789)
0809 03:32:36.668151 (+ 5779us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.668166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:36.668272 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:36.669465 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:36.669468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:36.669486 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:36.669592 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.669599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.670655 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.670659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.671867 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:36.671880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.672854 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.672858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:36.672962 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.672984 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:32:36.673455 (+ 471us) client_negotiation.cc:240] Negotiation successful
0809 03:32:36.673528 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5672,"thread_start_us":101,"threads_started":1}
I20260809 03:32:36.673873 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f529cec21a4cc9b21a846d4b31bc6f" candidate_uuid: "c42f019b277b493eb607730d7ba652bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" is_pre_election: true
I20260809 03:32:36.674242 4757 raft_consensus.cc:2454] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f019b277b493eb607730d7ba652bf in term 0.
I20260809 03:32:36.674266 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f529cec21a4cc9b21a846d4b31bc6f" candidate_uuid: "c42f019b277b493eb607730d7ba652bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" is_pre_election: true
I20260809 03:32:36.674563 4339 raft_consensus.cc:2454] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f019b277b493eb607730d7ba652bf in term 0.
I20260809 03:32:36.674784 4441 leader_election.cc:302] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c42f019b277b493eb607730d7ba652bf, e6a239474b5f40ed9b748a1fe53c7b7a; no voters:
I20260809 03:32:36.675096 4878 raft_consensus.cc:2781] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:32:36.675201 4878 raft_consensus.cc:482] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:32:36.675248 4878 raft_consensus.cc:3037] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:36.676558 4878 raft_consensus.cc:504] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.676918 4878 leader_election.cc:288] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [CANDIDATE]: Term 1 election: Requested vote from peers e6a239474b5f40ed9b748a1fe53c7b7a (127.15.222.67:33753), 34ea29ae21d34ad29d9a5fce95677fd4 (127.15.222.65:33429)
I20260809 03:32:36.677523 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f529cec21a4cc9b21a846d4b31bc6f" candidate_uuid: "c42f019b277b493eb607730d7ba652bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a"
I20260809 03:32:36.677850 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f529cec21a4cc9b21a846d4b31bc6f" candidate_uuid: "c42f019b277b493eb607730d7ba652bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ea29ae21d34ad29d9a5fce95677fd4"
I20260809 03:32:36.677966 4339 raft_consensus.cc:3037] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:36.679225 4339 raft_consensus.cc:2454] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f019b277b493eb607730d7ba652bf in term 1.
I20260809 03:32:36.677646 4757 raft_consensus.cc:3037] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:36.679566 4443 leader_election.cc:302] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ea29ae21d34ad29d9a5fce95677fd4, c42f019b277b493eb607730d7ba652bf; no voters:
I20260809 03:32:36.679724 4878 raft_consensus.cc:2781] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:36.679934 4878 raft_consensus.cc:686] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 1 LEADER]: Becoming Leader. State: Replica: c42f019b277b493eb607730d7ba652bf, State: Running, Role: LEADER
I20260809 03:32:36.680297 4878 consensus_queue.cc:237] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } }
I20260809 03:32:36.680805 4757 raft_consensus.cc:2454] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f019b277b493eb607730d7ba652bf in term 1.
I20260809 03:32:36.683990 4109 catalog_manager.cc:5495] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf reported cstate change: term changed from 0 to 1, leader changed from <none> to c42f019b277b493eb607730d7ba652bf (127.15.222.66). New cstate: current_term: 1 leader_uuid: "c42f019b277b493eb607730d7ba652bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f019b277b493eb607730d7ba652bf" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 } health_report { overall_health: UNKNOWN } } }
I20260809 03:32:36.716347 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.710917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.15.222.66:37829)
0809 03:32:36.711033 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:36.711038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:36.711270 (+ 232us) server_negotiation.cc:407] Connection header received
0809 03:32:36.711375 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:36.711378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:36.711445 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:36.711539 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:36.711898 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.712701 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.714019 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:36.714376 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:36.714473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:36.714881 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:36.715479 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:36.715591 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:36.716116 (+ 525us) server_negotiation.cc:299] Negotiation successful
0809 03:32:36.716176 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 03:32:36.728899 4339 raft_consensus.cc:1261] T 66f529cec21a4cc9b21a846d4b31bc6f P 34ea29ae21d34ad29d9a5fce95677fd4 [term 1 FOLLOWER]: Refusing update from remote peer c42f019b277b493eb607730d7ba652bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:36.738109 4884 consensus_queue.cc:1035] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ea29ae21d34ad29d9a5fce95677fd4" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:36.738874 4757 raft_consensus.cc:1261] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 1 FOLLOWER]: Refusing update from remote peer c42f019b277b493eb607730d7ba652bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:36.739527 4884 consensus_queue.cc:1035] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a239474b5f40ed9b748a1fe53c7b7a" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:36.750099 4887 mvcc.cc:205] Tried to move back new op lower bound from 7316465077153656832 to 7316465076966133760. Current Snapshot: MvccSnapshot[applied={T|T < 7316465077153656832}]
I20260809 03:32:36.756053 4888 mvcc.cc:205] Tried to move back new op lower bound from 7316465077153656832 to 7316465076966133760. Current Snapshot: MvccSnapshot[applied={T|T < 7316465077153656832}]
I20260809 03:32:36.758386 4889 mvcc.cc:205] Tried to move back new op lower bound from 7316465077153656832 to 7316465076966133760. Current Snapshot: MvccSnapshot[applied={T|T < 7316465077153656832}]
I20260809 03:32:36.768095 4112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:36.768277 4112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:36.771247 4112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffac365c698c4d338154e42fd75f01dd: Sending DeleteTablet for 3 replicas of tablet 66f529cec21a4cc9b21a846d4b31bc6f
I20260809 03:32:36.772723 4496 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:36 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58014
I20260809 03:32:36.773252 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:36 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:36488
I20260809 03:32:36.773739 4288 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f529cec21a4cc9b21a846d4b31bc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:36 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58932
I20260809 03:32:36.775110 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 4169
I20260809 03:32:36.788229 4904 tablet_replica.cc:330] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: stopping tablet replica
I20260809 03:32:36.788559 4905 tablet_replica.cc:330] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: stopping tablet replica
I20260809 03:32:36.806109 4905 raft_consensus.cc:2229] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:32:36.806593 4904 raft_consensus.cc:2229] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:32:36.807199 4904 raft_consensus.cc:2258] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:36.807696 4905 raft_consensus.cc:2258] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:36.809561 4904 ts_tablet_manager.cc:1905] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:36.810612 4905 ts_tablet_manager.cc:1905] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:36.813618 4905 ts_tablet_manager.cc:1918] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:36.816339 4904 ts_tablet_manager.cc:1918] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:36.816493 4904 log.cc:1196] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/66f529cec21a4cc9b21a846d4b31bc6f
I20260809 03:32:36.816913 4904 ts_tablet_manager.cc:1939] T 66f529cec21a4cc9b21a846d4b31bc6f P c42f019b277b493eb607730d7ba652bf: Deleting consensus metadata
I20260809 03:32:36.817071 4905 log.cc:1196] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/66f529cec21a4cc9b21a846d4b31bc6f
I20260809 03:32:36.817629 4905 ts_tablet_manager.cc:1939] T 66f529cec21a4cc9b21a846d4b31bc6f P e6a239474b5f40ed9b748a1fe53c7b7a: Deleting consensus metadata
I20260809 03:32:36.819020 4090 catalog_manager.cc:4841] TS c42f019b277b493eb607730d7ba652bf (127.15.222.66:37829): tablet 66f529cec21a4cc9b21a846d4b31bc6f (table test-table [id=b9136f38d8554c5d8040fc29826a8209]) successfully deleted
I20260809 03:32:36.820403 4089 catalog_manager.cc:4841] TS e6a239474b5f40ed9b748a1fe53c7b7a (127.15.222.67:33753): tablet 66f529cec21a4cc9b21a846d4b31bc6f (table test-table [id=b9136f38d8554c5d8040fc29826a8209]) successfully deleted
W20260809 03:32:36.825698 4092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:33429 (error 108)
I20260809 03:32:36.826578 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 4415
I20260809 03:32:36.826864 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:36.826543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:33429 (local address 127.0.0.1:58944)
0809 03:32:36.826717 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:36.826783 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260809 03:32:36.832893 4092 catalog_manager.cc:4584] TS 34ea29ae21d34ad29d9a5fce95677fd4 (127.15.222.65:33429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66f529cec21a4cc9b21a846d4b31bc6f: Network error: Client connection negotiation failed: client connection to 127.15.222.65:33429: connect: Connection refused (error 111)
I20260809 03:32:36.850921 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 4605
I20260809 03:32:36.868988 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 4055
I20260809 03:32:36.880431 16249 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 03:32:37.225857 16249 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 03:32:37.225960 16249 mini_ranger.cc:66] Stopping Ranger...
I20260809 03:32:37.569921 16249 mini_ranger.cc:68] Stopped Ranger
2026-08-09 03:32:37.570 UTC [2741] LOG: received smart shutdown request
2026-08-09 03:32:37.574 UTC [2741] LOG: background worker "logical replication launcher" (PID 2751) exited with exit code 1
2026-08-09 03:32:37.574 UTC [2745] LOG: shutting down
2026-08-09 03:32:37.696 UTC [2741] LOG: database system is shut down
2026-08-09T03:32:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45773 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:37 dist-test-slave-cvwc krb5kdc[4929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:37 dist-test-slave-cvwc krb5kdc[4929](info): set up 2 sockets
Aug 09 03:32:37 dist-test-slave-cvwc krb5kdc[4929](info): commencing operation
krb5kdc: starting...
W20260809 03:32:38.939944 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:32:39.054147 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37785
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:39.163455 4947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:39.163697 4947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:39.163735 4947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:39.167636 4947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:39.167687 4947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:39.167706 4947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:39.167721 4947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:39.167735 4947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:39.172855 4947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:39.173844 4947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:39.174827 4947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:39.182893 4956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.182875 4952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.182875 4954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:39.183025 4947 server_base.cc:1034] running on GCE node
I20260809 03:32:39.183681 4947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:39.184742 4947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:39.185902 4947 hybrid_clock.cc:648] HybridClock initialized: now 1786246359185888 us; error 30 us; skew 500 ppm
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:39.188421 4947 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:39.189277 4947 webserver.cc:460] Webserver started at http://127.15.222.126:43907/ using document root <none> and password file <none>
I20260809 03:32:39.189831 4947 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:39.189900 4947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:39.190099 4947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:39.192060 4947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "bcf936f9e48a43a2a91cb1e482f50a48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "3610fdd3ea8bd17a376b97c5e72b10e8"
server_key_iv: "07490c129bb8210bacb9bde0d71ef9ec"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.192600 4947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "bcf936f9e48a43a2a91cb1e482f50a48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "3610fdd3ea8bd17a376b97c5e72b10e8"
server_key_iv: "07490c129bb8210bacb9bde0d71ef9ec"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.196110 4947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:32:39.198345 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:39.199501 4947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:39.199611 4947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "bcf936f9e48a43a2a91cb1e482f50a48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "3610fdd3ea8bd17a376b97c5e72b10e8"
server_key_iv: "07490c129bb8210bacb9bde0d71ef9ec"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.199699 4947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:39.212980 4947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:39.215634 4947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:39.215809 4947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:39.222373 4947 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:37785
I20260809 03:32:39.222644 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:37785 every 8 connection(s)
I20260809 03:32:39.223254 4947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:39.225908 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:39.229417 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 4947
I20260809 03:32:39.229545 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:32:39.229856 16249 external_mini_cluster.cc:1428] Setting key 1c3ad7f9c0a1fb501d41bdefcd013ac2
I20260809 03:32:39.229591 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.125@KRBTEST.COM" created.
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:39.246752 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.232062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.15.222.126:37785)
0809 03:32:39.232508 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.232528 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.232825 (+ 297us) server_negotiation.cc:407] Connection header received
0809 03:32:39.233974 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.233979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.234154 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.234300 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.235316 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.236150 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.237524 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.237904 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.238012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.242522 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.242528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.242538 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.242558 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.243963 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.244376 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.244381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.244382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.244485 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.244763 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.244766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.244767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.245096 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.245210 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.245546 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.245796 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260809 03:32:39.246752 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.232062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37785 (local address 127.0.0.1:44094)
0809 03:32:39.232658 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.232706 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.233641 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.234353 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.234358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.234830 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.235169 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.235189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.236278 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.236282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.237418 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.237431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.238049 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.238054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.238180 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.238693 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.238714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.242389 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.244060 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.244068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.244077 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.244280 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.244568 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.244571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.244573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.244674 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.245201 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.245207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.245330 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.245470 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.245852 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":129,"spinlock_wait_cycles":12544,"thread_start_us":161,"threads_started":1}
I20260809 03:32:39.248957 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:39.249580 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.249358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52698)
0809 03:32:39.249451 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.249515 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 03:32:39.249797 4970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
I20260809 03:32:39.250556 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.250363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52714)
0809 03:32:39.250453 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.250495 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 03:32:39.250682 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45013: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
I20260809 03:32:39.255884 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:45013
--webserver_interface=127.15.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:39.299921 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } attempt: 1
I20260809 03:32:39.300796 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.300510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52724)
0809 03:32:39.300657 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.300717 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 03:32:39.301450 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.301267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52726)
0809 03:32:39.301348 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.301388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:32:39.301568 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45013: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
W20260809 03:32:39.363742 5057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:39.363996 5057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:39.364033 5057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:39.367934 5057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:39.367990 5057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:39.368007 5057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:39.368024 5057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:39.368052 5057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:39.369742 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } attempt: 2
I20260809 03:32:39.370548 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.370270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52738)
0809 03:32:39.370409 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.370471 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:32:39.371194 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.371006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52748)
0809 03:32:39.371094 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.371138 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 03:32:39.371300 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45013: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45013: connect: Connection refused (error 111)
I20260809 03:32:39.373517 5057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:39.374548 5057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:39.375515 5057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:39.383771 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.383773 5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.383771 5067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:39.384074 5057 server_base.cc:1034] running on GCE node
I20260809 03:32:39.384599 5057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:39.385603 5057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:39.386758 5057 hybrid_clock.cc:648] HybridClock initialized: now 1786246359386717 us; error 50 us; skew 500 ppm
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:39.389397 5057 init.cc:377] Logged in from keytab as kudu/127.15.222.125@KRBTEST.COM (short username kudu)
I20260809 03:32:39.390236 5057 webserver.cc:460] Webserver started at http://127.15.222.125:38749/ using document root <none> and password file <none>
I20260809 03:32:39.390764 5057 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:39.390828 5057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:39.391026 5057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:39.392972 5057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data/instance:
uuid: "9a4cd62a39b64af18a3204a44decd612"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "81bd1049f312c8d7a13b173b801cae32"
server_key_iv: "e0af2776ba49228a03e2a7c7bf01a4e1"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.393507 5057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal/instance:
uuid: "9a4cd62a39b64af18a3204a44decd612"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "81bd1049f312c8d7a13b173b801cae32"
server_key_iv: "e0af2776ba49228a03e2a7c7bf01a4e1"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.396944 5057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 03:32:39.399230 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:39.400345 5057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:32:39.400465 5057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "9a4cd62a39b64af18a3204a44decd612"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "81bd1049f312c8d7a13b173b801cae32"
server_key_iv: "e0af2776ba49228a03e2a7c7bf01a4e1"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.400553 5057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:39.409638 5057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:39.412231 5057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:39.412374 5057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:39.418922 5057 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.125:45013
I20260809 03:32:39.419193 5155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.125:45013 every 8 connection(s)
I20260809 03:32:39.419803 5057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
I20260809 03:32:39.421780 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5057
I20260809 03:32:39.421895 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-1/wal/instance
I20260809 03:32:39.422137 16249 external_mini_cluster.cc:1428] Setting key ab973a63d938e2fd8b113d11aa368418
I20260809 03:32:39.422533 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:39.427000 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.124@KRBTEST.COM" created.
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:39.440233 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.428996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.15.222.126:37785)
0809 03:32:39.429112 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.429115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.430135 (+ 1020us) server_negotiation.cc:407] Connection header received
0809 03:32:39.430998 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.431001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.431064 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.431134 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.432256 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.432790 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.433992 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.434233 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.434324 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.436774 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.436778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.436781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.436811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.438307 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.438787 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.438790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.438791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.438868 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.439172 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.439175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.439177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.439358 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.439454 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.439942 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.440070 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:32:39.440984 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.429490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37785 (local address 127.0.0.1:44100)
0809 03:32:39.429981 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.430033 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.430798 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.431304 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.431313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.431635 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.432100 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.432112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.432908 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.432910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.433868 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.433878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.434369 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.434373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.434546 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.435078 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.435094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.436602 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.438412 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.438417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.438428 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.438682 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.438953 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.438956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.438958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.439085 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.439465 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.439472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.439706 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.439865 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.440181 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":151,"threads_started":1}
I20260809 03:32:39.442315 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
I20260809 03:32:39.448271 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:41309
--webserver_interface=127.15.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:39.450539 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.442761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52760)
0809 03:32:39.442875 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.442887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.442961 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.443478 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.443481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.443640 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.443762 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.443769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.444704 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.444706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.445314 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.445322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.445878 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.445890 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.445963 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.446398 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.446409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.447563 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.449172 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.449175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.449177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.449345 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.449591 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.449594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.449596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.449667 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.450083 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.450086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.450166 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.450252 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.450369 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:32:39.450588 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.442850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.15.222.125:45013)
0809 03:32:39.443166 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.443175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.443192 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:32:39.443224 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.443228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.443365 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.443481 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.443852 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.444604 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.445410 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.445796 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.445870 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.447671 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.447679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.447689 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.447709 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.449077 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.449439 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.449443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.449445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.449514 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.449757 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.449761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.449762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.449990 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.450105 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.450338 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.450455 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":153,"threads_started":1}
I20260809 03:32:39.452414 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:39.453037 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.452810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33054)
0809 03:32:39.452910 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.452980 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 03:32:39.453207 5089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
I20260809 03:32:39.454017 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.453826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33056)
0809 03:32:39.453922 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.453967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:32:39.454130 5157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
I20260809 03:32:39.462458 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } attempt: 3
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
I20260809 03:32:39.470690 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.462903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52762)
0809 03:32:39.463019 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.463033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.463112 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.463326 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.463328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.463477 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.463580 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.463585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.464538 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.464540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.465465 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.465475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.466091 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.466094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.466176 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.466623 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.466634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.467804 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.469435 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.469438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.469440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.469620 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.469883 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.469886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.469887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.469958 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.470305 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.470307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.470388 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.470475 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.470574 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:32:39.470780 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.462949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.15.222.125:45013)
0809 03:32:39.463060 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.463064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.463079 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:32:39.463187 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.463190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.463233 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.463311 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.463669 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.464453 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.465569 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.465992 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.466092 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.467927 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.467931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.467934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.467960 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.469328 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.469701 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.469705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.469706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.469810 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.470033 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.470035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.470037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.470229 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.470319 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.470534 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.470642 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:32:39.471979 5047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:39.472623 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.472410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33066)
0809 03:32:39.472518 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.472566 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:32:39.473344 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.473119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33078)
0809 03:32:39.473258 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.473299 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260809 03:32:39.473480 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
I20260809 03:32:39.503357 5157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 1
I20260809 03:32:39.504175 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.503869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33086)
0809 03:32:39.504015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.504084 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 03:32:39.504609 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 1
I20260809 03:32:39.504925 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.504687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33102)
0809 03:32:39.504783 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.504833 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:32:39.505144 5157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
I20260809 03:32:39.505230 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.504982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33110)
0809 03:32:39.505105 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.505156 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 03:32:39.505841 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.505628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33122)
0809 03:32:39.505743 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.505788 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:32:39.505963 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
W20260809 03:32:39.559659 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:39.559906 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:39.559944 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:39.563799 5168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:39.563848 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:39.563865 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:39.563889 5168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:39.563916 5168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:39.568934 5168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37785,127.15.222.125:45013,127.15.222.124:41309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:41309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:39.569919 5168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:39.570889 5168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:32:39.573326 5157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 2
I20260809 03:32:39.574129 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.573855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33138)
0809 03:32:39.573990 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.574052 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 03:32:39.574891 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.574656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33146)
0809 03:32:39.574772 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.574824 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 03:32:39.575060 5157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
W20260809 03:32:39.579180 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.579182 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:39.579180 5182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:39.579428 5168 server_base.cc:1034] running on GCE node
I20260809 03:32:39.580044 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:39.581125 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:39.581100 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 2
I20260809 03:32:39.581779 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.581507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33156)
0809 03:32:39.581626 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.581676 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 03:32:39.582304 5168 hybrid_clock.cc:648] HybridClock initialized: now 1786246359582286 us; error 30 us; skew 500 ppm
I20260809 03:32:39.582382 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.582201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33162)
0809 03:32:39.582288 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.582328 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260809 03:32:39.582535 5047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:41309: Network error: Client connection negotiation failed: client connection to 127.15.222.124:41309: connect: Connection refused (error 111)
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:39.584882 5168 init.cc:377] Logged in from keytab as kudu/127.15.222.124@KRBTEST.COM (short username kudu)
I20260809 03:32:39.585743 5168 webserver.cc:460] Webserver started at http://127.15.222.124:46001/ using document root <none> and password file <none>
I20260809 03:32:39.586264 5168 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:39.586328 5168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:39.586524 5168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:39.588466 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data/instance:
uuid: "91c1dadac6c34a2b9f2ba8da1be82b48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "fe938e2a231813dfc0a7853f31ea17e4"
server_key_iv: "f1f176fbeb9f2696239adc3a30a0a6f5"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.589004 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal/instance:
uuid: "91c1dadac6c34a2b9f2ba8da1be82b48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "fe938e2a231813dfc0a7853f31ea17e4"
server_key_iv: "f1f176fbeb9f2696239adc3a30a0a6f5"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.592520 5168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:32:39.594837 5197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:39.595992 5168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:32:39.596102 5168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "91c1dadac6c34a2b9f2ba8da1be82b48"
format_stamp: "Formatted at 2026-08-09 03:32:39 on dist-test-slave-cvwc"
server_key: "fe938e2a231813dfc0a7853f31ea17e4"
server_key_iv: "f1f176fbeb9f2696239adc3a30a0a6f5"
server_key_version: "encryptionkey@0"
I20260809 03:32:39.596192 5168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:39.619509 5168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:39.622193 5168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:39.622368 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:39.629069 5168 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.124:41309
I20260809 03:32:39.629320 5279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.124:41309 every 8 connection(s)
I20260809 03:32:39.630012 5168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
I20260809 03:32:39.632634 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:39.634438 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5168
I20260809 03:32:39.634557 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786246200571663-16249-0/minicluster-data/master-2/wal/instance
I20260809 03:32:39.634804 16249 external_mini_cluster.cc:1428] Setting key d4b9a400093239f5ea8daf151bc03dce
I20260809 03:32:39.636567 5281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:39.647464 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.638861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.15.222.126:37785)
0809 03:32:39.638976 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.638980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.639287 (+ 307us) server_negotiation.cc:407] Connection header received
0809 03:32:39.639419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.639421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.639465 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.639526 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.640211 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.640708 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.641742 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.641967 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.642057 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.644369 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.644373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.644376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.644404 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.645857 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.646260 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.646264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.646266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.646345 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.646584 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.646587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.646589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.646772 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.646855 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.647188 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.647303 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:32:39.648234 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.638805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37785 (local address 127.0.0.1:44116)
0809 03:32:39.639171 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.639207 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.639321 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.639584 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.639589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.639937 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.640084 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.640092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.640829 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.640832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.641595 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.641606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.642102 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.642106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.642203 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.642662 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.642677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.644212 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.645951 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.645960 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.645969 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.646170 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.646429 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.646432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.646434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.646507 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.646869 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.646875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.646994 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.647130 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.647474 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":126,"thread_start_us":157,"threads_started":1}
I20260809 03:32:39.648236 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.635978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.15.222.124:41309)
0809 03:32:39.636329 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.636344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.636377 (+ 33us) server_negotiation.cc:407] Connection header received
0809 03:32:39.636997 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.637014 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.637326 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.637609 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.638497 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.639776 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.640695 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.641221 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.641330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.643856 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.643863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.643875 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.643898 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.645614 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.646033 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.646037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.646039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.646142 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.646399 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.646402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.646403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.646738 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.646904 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.647138 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.647411 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260809 03:32:39.650271 5281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:39.658329 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.650730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45013 (local address 127.0.0.1:52764)
0809 03:32:39.650968 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.650981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.651047 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.651334 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.651337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.651502 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.651590 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.651597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.652443 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.652445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.652939 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.652949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.653349 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.653353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.653445 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.653860 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.653871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.655063 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.656883 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.656886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.656888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.657144 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.657421 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.657423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.657425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.657518 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.657909 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.657912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.658008 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.658113 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.658216 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260809 03:32:39.658424 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.651004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.15.222.125:45013)
0809 03:32:39.651121 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.651125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.651139 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:32:39.651182 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.651184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.651228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.651302 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.651757 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.652328 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.653055 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.653260 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.653328 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.655194 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.655197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.655199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.655222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.656772 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.657235 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.657238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.657240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.657330 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.657601 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.657603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.657604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.657799 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.657873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.658172 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.658278 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:32:39.659001 5281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:32:39.659930 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.651458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.15.222.126:37785)
0809 03:32:39.651563 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.651567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.651579 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:39.651610 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.651613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.651649 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.651711 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.652370 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.652851 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.653707 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.654006 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.654080 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.656503 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.656506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.656508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.656529 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.658000 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.658715 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.658718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.658719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.658794 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.659084 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.659087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.659088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.659321 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.659528 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.659674 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.659786 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
I20260809 03:32:39.666415 5157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 3
I20260809 03:32:39.667021 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.659445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33186)
0809 03:32:39.659531 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.659544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.659613 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.659954 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.659957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.660102 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.660181 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.660188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.661389 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.661392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.661944 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.661953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.662357 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.662360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.662464 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.662831 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.662840 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.663950 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.665483 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.665486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.665488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.665741 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.666008 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.666011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.666013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.666105 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.666558 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.666562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.666655 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.666759 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.666872 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":15}
I20260809 03:32:39.667459 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.659604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.15.222.124:41309)
0809 03:32:39.659695 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.659698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.659708 (+ 10us) server_negotiation.cc:407] Connection header received
0809 03:32:39.659737 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.659740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.659800 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.659871 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.660290 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.661310 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.662044 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.662267 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.662349 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.664073 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.664075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.664077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.664097 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.665368 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.665832 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.665835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.665837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.665923 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.666204 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.666207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.666209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.666421 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.666512 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.667189 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.667316 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":36}
I20260809 03:32:39.669554 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.660980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.15.222.125:45013)
0809 03:32:39.661091 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.661095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.661108 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:32:39.661165 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.661167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.661205 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.661281 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.661783 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.662229 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.663022 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.663231 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.663313 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.665496 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.665499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.665502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.665523 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.667940 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.668445 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.668448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.668450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.668527 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.668765 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.668768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.668770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.668931 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.669218 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.669313 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.669418 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:32:39.670629 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: Bootstrap starting.
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:32:39.673197 5281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:39.673417 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:39.674094 5281 log.cc:824] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:39.675195 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.666744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33188)
0809 03:32:39.667086 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.667099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.667170 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.667397 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.667401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.667613 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.667699 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.667705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.668529 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.668533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.669262 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.669274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.670067 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.670071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.670150 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.670528 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.670541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.671760 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.673590 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.673593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.673595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.673855 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.674152 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.674155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.674157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.674271 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.674731 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.674734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.674849 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.674934 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.675057 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260809 03:32:39.675742 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.666825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.15.222.124:41309)
0809 03:32:39.667127 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.667136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.667152 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:32:39.667250 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.667253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.667291 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.667365 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.667811 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.668388 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.669413 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.669940 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.670337 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.671896 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.671899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.671902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.671925 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.673464 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.673948 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.673952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.673953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.674055 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.674367 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.674370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.674372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.674604 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.675467 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.675538 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.675625 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
I20260809 03:32:39.676040 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: No bootstrap required, opened a new log
I20260809 03:32:39.678620 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: Bootstrap starting.
I20260809 03:32:39.679967 5281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.680217 5281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:39.680272 5281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c1dadac6c34a2b9f2ba8da1be82b48, State: Initialized, Role: FOLLOWER
I20260809 03:32:39.680728 5281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.681200 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.681265 5157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:39.681324 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.681674 5047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } attempt: 3
I20260809 03:32:39.682118 5281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:39.682142 5157 log.cc:824] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:32:39 dist-test-slave-cvwc krb5kdc[4929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
W20260809 03:32:39.687218 5318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:39.687282 5318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:39.685650 5168 master_runner.cc:186] Error getting master registration for 127.15.222.126:37785: OK, instance_id { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" instance_seqno: 1786246359220590 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:39.688732 5168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:39.691318 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: No bootstrap required, opened a new log
I20260809 03:32:39.692236 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.682252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.15.222.124:41309)
0809 03:32:39.682349 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.682353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.682367 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:32:39.682401 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.682404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.682442 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.682516 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.683190 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.683950 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.684934 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.685300 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.685402 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.687784 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.687787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.687790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.687812 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.689950 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.690424 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.690427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.690429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.690511 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.690813 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.690816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.690818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.691019 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.691110 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.691970 (+ 860us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.692088 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:32:39.692673 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.682083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:41309 (local address 127.0.0.1:33202)
0809 03:32:39.682203 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:39.682217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:39.682329 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:39.682640 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:39.682643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:39.682849 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:39.682965 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.682972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.684056 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.684059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.684801 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:39.684812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.685519 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.685522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:39.685605 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.686142 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:39.686156 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:39.687653 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:39.690066 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.690070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.690072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.690315 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.690616 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:39.690619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:39.690621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:39.690713 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:39.691159 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:39.691162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:39.691261 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 03:32:39.691485 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 03:32:39.691617 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 03:32:39.695585 5157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.695770 5157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:39.695844 5157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4cd62a39b64af18a3204a44decd612, State: Initialized, Role: FOLLOWER
I20260809 03:32:39.696281 5157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.696334 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: Bootstrap starting.
I20260809 03:32:39.696734 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.696920 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.697515 5157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:39.699077 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:39.702268 5047 log.cc:824] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:39.702446 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.682973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.15.222.125:45013)
0809 03:32:39.683135 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.683139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.684323 (+ 1184us) server_negotiation.cc:407] Connection header received
0809 03:32:39.684486 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.684489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.684540 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.684584 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.685451 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.686168 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.687293 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.687599 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.687674 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.689336 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.689340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.689343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.689368 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.698572 (+ 9204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.701063 (+ 2491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.701068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.701070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.701158 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.701453 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.701457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.701458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.701664 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.701789 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.702178 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.702297 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 03:32:39.704622 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.678052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.15.222.126:37785)
0809 03:32:39.678161 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.678166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.683244 (+ 5078us) server_negotiation.cc:407] Connection header received
0809 03:32:39.683344 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.683347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.683391 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.683466 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.688999 (+ 5533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.689741 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.690825 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.691153 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.691229 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.692928 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.692931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.692934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.692960 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.699834 (+ 6874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.703172 (+ 3338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.703176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.703178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.703264 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.703563 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.703566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.703568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.703790 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.703910 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.704346 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.704479 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:32:39.701934 5057 master_runner.cc:186] Error getting master registration for 127.15.222.126:37785: OK, instance_id { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" instance_seqno: 1786246359220590 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:32:39.705511 5057 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:32:39.706796 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: No bootstrap required, opened a new log
I20260809 03:32:39.709969 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:39.683121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.15.222.124:41309)
0809 03:32:39.684104 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:39.684107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:39.698107 (+ 14000us) server_negotiation.cc:407] Connection header received
0809 03:32:39.698316 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:39.698320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:39.698367 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:39.698416 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:39.699381 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.700128 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.701307 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:39.701601 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:39.701670 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:39.707202 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:39.707205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:39.707208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:39.707234 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:39.708481 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.708850 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.708853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.708855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.708928 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:39.709164 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:39.709167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:39.709169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:39.709333 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:39.709438 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:39.709668 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 03:32:39.709818 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919}
I20260809 03:32:39.710776 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.710985 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:39.711040 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf936f9e48a43a2a91cb1e482f50a48, State: Initialized, Role: FOLLOWER
I20260809 03:32:39.711493 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.711944 5339 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.712143 5339 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.712759 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 03:32:39.716814 5336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:39.716883 5336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 03:32:39.717653 4975 tablet.cc:2369] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:32:39.719470 5353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:32:39.719535 5353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:32:39.721005 4947 master_runner.cc:263] No leader master found from master bcf936f9e48a43a2a91cb1e482f50a48
I20260809 03:32:39.721071 4947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 03:32:39.873673 5204 tablet.cc:2369] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:39.899005 5321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:39.899261 5321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.900408 5321 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf936f9e48a43a2a91cb1e482f50a48 (127.15.222.126:37785), 91c1dadac6c34a2b9f2ba8da1be82b48 (127.15.222.124:41309)
I20260809 03:32:39.900843 5004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a4cd62a39b64af18a3204a44decd612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" is_pre_election: true
I20260809 03:32:39.900856 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a4cd62a39b64af18a3204a44decd612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" is_pre_election: true
I20260809 03:32:39.901151 5004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4cd62a39b64af18a3204a44decd612 in term 0.
I20260809 03:32:39.901185 5241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4cd62a39b64af18a3204a44decd612 in term 0.
I20260809 03:32:39.901525 5084 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4cd62a39b64af18a3204a44decd612, bcf936f9e48a43a2a91cb1e482f50a48; no voters:
I20260809 03:32:39.901911 5321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:32:39.901999 5321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:32:39.902043 5321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:39.903345 5321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.903677 5321 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [CANDIDATE]: Term 1 election: Requested vote from peers bcf936f9e48a43a2a91cb1e482f50a48 (127.15.222.126:37785), 91c1dadac6c34a2b9f2ba8da1be82b48 (127.15.222.124:41309)
I20260809 03:32:39.904028 5004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a4cd62a39b64af18a3204a44decd612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf936f9e48a43a2a91cb1e482f50a48"
I20260809 03:32:39.904148 5004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:39.904147 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a4cd62a39b64af18a3204a44decd612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48"
I20260809 03:32:39.904300 5241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:39.905284 5004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4cd62a39b64af18a3204a44decd612 in term 1.
I20260809 03:32:39.905428 5241 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4cd62a39b64af18a3204a44decd612 in term 1.
I20260809 03:32:39.905618 5084 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4cd62a39b64af18a3204a44decd612, bcf936f9e48a43a2a91cb1e482f50a48; no voters:
I20260809 03:32:39.905817 5321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:39.906036 5321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4cd62a39b64af18a3204a44decd612, State: Running, Role: LEADER
I20260809 03:32:39.906358 5321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } }
I20260809 03:32:39.908298 5356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4cd62a39b64af18a3204a44decd612. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.908416 5356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:39.908794 5358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:39.909793 5358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 03:32:39.913746 5091 tablet.cc:2369] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:39.914685 5004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [term 1 FOLLOWER]: Refusing update from remote peer 9a4cd62a39b64af18a3204a44decd612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:39.914723 5241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [term 1 FOLLOWER]: Refusing update from remote peer 9a4cd62a39b64af18a3204a44decd612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:39.915369 5356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:39.915532 5321 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:39.920320 5339 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4cd62a39b64af18a3204a44decd612. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.920454 5339 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.920428 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4cd62a39b64af18a3204a44decd612. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.920545 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.922217 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.922348 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:39.922479 5339 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.922590 5339 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.922963 5356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.923070 5356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612 [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:39.923897 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a4cd62a39b64af18a3204a44decd612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf936f9e48a43a2a91cb1e482f50a48" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37785 } } peers { permanent_uuid: "9a4cd62a39b64af18a3204a44decd612" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45013 } } peers { permanent_uuid: "91c1dadac6c34a2b9f2ba8da1be82b48" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 41309 } } }
I20260809 03:32:39.924006 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:32:39.924803 5358 catalog_manager.cc:1347] Generated new cluster ID: 136450fdb7af479bad762189ff24cb41
I20260809 03:32:39.924856 5358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:39.933485 5358 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:39.934820 5358 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:39.939740 5358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4cd62a39b64af18a3204a44decd612: Generated new TSK 0
I20260809 03:32:39.940753 5358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:40.688567 5318 catalog_manager.cc:1259] Loaded cluster ID: 136450fdb7af479bad762189ff24cb41
I20260809 03:32:40.688654 5318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: loading cluster ID for follower catalog manager: success
I20260809 03:32:40.690407 5318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: acquiring CA information for follower catalog manager: success
I20260809 03:32:40.691728 5318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91c1dadac6c34a2b9f2ba8da1be82b48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:40.720676 5353 catalog_manager.cc:1259] Loaded cluster ID: 136450fdb7af479bad762189ff24cb41
I20260809 03:32:40.720746 5353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: loading cluster ID for follower catalog manager: success
I20260809 03:32:40.722361 5353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: acquiring CA information for follower catalog manager: success
I20260809 03:32:40.723491 5353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcf936f9e48a43a2a91cb1e482f50a48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:32:41.015101 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 4947
I20260809 03:32:41.027896 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5057
I20260809 03:32:41.038084 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5168
2026-08-09T03:32:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3250 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:41 dist-test-slave-cvwc krb5kdc[5378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:41 dist-test-slave-cvwc krb5kdc[5378](info): set up 2 sockets
Aug 09 03:32:41 dist-test-slave-cvwc krb5kdc[5378](info): commencing operation
krb5kdc: starting...
W20260809 03:32:42.177665 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:32:42.292187 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43809
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:42.401680 5393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:42.401958 5393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:42.402005 5393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:42.405936 5393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:42.405992 5393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:42.406018 5393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:42.406054 5393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:42.406085 5393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:42.411162 5393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:43809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:42.412096 5393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:42.413231 5393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:42.421100 5403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:42.421136 5398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:42.421262 5393 server_base.cc:1034] running on GCE node
W20260809 03:32:42.421108 5407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:42.421929 5393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:42.422930 5393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:42.424086 5393 hybrid_clock.cc:648] HybridClock initialized: now 1786246362424073 us; error 24 us; skew 500 ppm
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246362, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:42.426682 5393 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:42.427520 5393 webserver.cc:460] Webserver started at http://127.15.222.126:42759/ using document root <none> and password file <none>
I20260809 03:32:42.428023 5393 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:42.428088 5393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:42.428283 5393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:42.430258 5393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "d58aae1d2c11418eb08c5f5a800ac78c"
format_stamp: "Formatted at 2026-08-09 03:32:42 on dist-test-slave-cvwc"
server_key: "6fa5e114e22b52c63f076491d45015f9"
server_key_iv: "21fd393280baf2762b22bbeae3d0f759"
server_key_version: "encryptionkey@0"
I20260809 03:32:42.430788 5393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "d58aae1d2c11418eb08c5f5a800ac78c"
format_stamp: "Formatted at 2026-08-09 03:32:42 on dist-test-slave-cvwc"
server_key: "6fa5e114e22b52c63f076491d45015f9"
server_key_iv: "21fd393280baf2762b22bbeae3d0f759"
server_key_version: "encryptionkey@0"
I20260809 03:32:42.434244 5393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:32:42.436486 5418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:42.437649 5393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:42.437798 5393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "d58aae1d2c11418eb08c5f5a800ac78c"
format_stamp: "Formatted at 2026-08-09 03:32:42 on dist-test-slave-cvwc"
server_key: "6fa5e114e22b52c63f076491d45015f9"
server_key_iv: "21fd393280baf2762b22bbeae3d0f759"
server_key_version: "encryptionkey@0"
I20260809 03:32:42.437891 5393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:42.450532 5393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:42.453365 5393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:42.453532 5393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:42.459816 5393 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:43809
I20260809 03:32:42.459878 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:43809 every 8 connection(s)
I20260809 03:32:42.460700 5393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:42.463392 5491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:42.467792 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5393
I20260809 03:32:42.467891 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:32:42.468144 16249 external_mini_cluster.cc:1428] Setting key 458fcb3ec80178ec152d4ebbfe7a3fd3
I20260809 03:32:42.468971 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c: Bootstrap starting.
I20260809 03:32:42.471592 5491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:42.472430 5491 log.cc:824] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:42.474434 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c: No bootstrap required, opened a new log
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:42.478106 5491 raft_consensus.cc:348] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } }
I20260809 03:32:42.478394 5491 raft_consensus.cc:374] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:42.478457 5491 raft_consensus.cc:729] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58aae1d2c11418eb08c5f5a800ac78c, State: Initialized, Role: FOLLOWER
I20260809 03:32:42.479010 5491 consensus_queue.cc:260] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } }
I20260809 03:32:42.479174 5491 raft_consensus.cc:388] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:42.479238 5491 raft_consensus.cc:482] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:42.479334 5491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:42.480443 5491 raft_consensus.cc:504] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } }
I20260809 03:32:42.480873 5491 leader_election.cc:302] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d58aae1d2c11418eb08c5f5a800ac78c; no voters:
I20260809 03:32:42.481218 5491 leader_election.cc:288] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:42.481312 5499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:42.481554 5499 raft_consensus.cc:686] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [term 1 LEADER]: Becoming Leader. State: Replica: d58aae1d2c11418eb08c5f5a800ac78c, State: Running, Role: LEADER
I20260809 03:32:42.481950 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } }
I20260809 03:32:42.482606 5491 sys_catalog.cc:564] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:42.482833 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:42.469400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.15.222.126:43809)
0809 03:32:42.469833 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:42.469851 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:42.469894 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:32:42.470700 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:42.470723 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:42.471126 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:42.471481 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:42.472136 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:42.473432 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:42.474293 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:42.474949 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:42.475109 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:42.477621 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:42.477627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:42.477644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:42.477675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:42.479658 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:42.480060 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:42.480066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:42.480072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:42.480224 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:42.480570 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:42.480574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:42.480575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:42.481091 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:42.481294 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:42.481582 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 03:32:42.481899 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260809 03:32:42.483134 5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58aae1d2c11418eb08c5f5a800ac78c. Latest consensus state: current_term: 1 leader_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } } }
I20260809 03:32:42.483276 5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:42.483654 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58aae1d2c11418eb08c5f5a800ac78c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43809 } } }
I20260809 03:32:42.483744 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:42.483999 5508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:42.485903 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:32:42.487195 5508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:42.492446 5508 catalog_manager.cc:1347] Generated new cluster ID: ccf9664e740b4f8aa9756031e659cb5a
I20260809 03:32:42.492506 5508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:42.503401 5508 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:42.504593 5508 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:42.521513 5508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d58aae1d2c11418eb08c5f5a800ac78c: Generated new TSK 0
I20260809 03:32:42.522147 5508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:32:42.541239 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5393
2026-08-09T03:32:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1504 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5537](info): set up 2 sockets
Aug 09 03:32:42 dist-test-slave-cvwc krb5kdc[5537](info): commencing operation
krb5kdc: starting...
W20260809 03:32:43.688679 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:32:43 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:32:43.804266 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:43.912374 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:43.912612 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:43.912667 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:43.916561 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:43.916611 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:43.916637 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:43.916658 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:43.916683 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:43.921793 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:43.922739 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:43.923695 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:32:43.931954 5557 server_base.cc:1034] running on GCE node
W20260809 03:32:43.931885 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:43.931911 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:43.931905 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:43.932655 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:43.933691 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:43.934887 5557 hybrid_clock.cc:648] HybridClock initialized: now 1786246363934867 us; error 25 us; skew 500 ppm
Aug 09 03:32:43 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246363, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:43.937394 5557 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:43.938249 5557 webserver.cc:460] Webserver started at http://127.15.222.126:42903/ using document root <none> and password file <none>
I20260809 03:32:43.938774 5557 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:43.938840 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:43.939057 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:43.941078 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a"
format_stamp: "Formatted at 2026-08-09 03:32:43 on dist-test-slave-cvwc"
server_key: "6f6602bf45e67b76a396c7ff335073b0"
server_key_iv: "c559e491833f06909f8b72bdae5aeaac"
server_key_version: "encryptionkey@0"
I20260809 03:32:43.941612 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a"
format_stamp: "Formatted at 2026-08-09 03:32:43 on dist-test-slave-cvwc"
server_key: "6f6602bf45e67b76a396c7ff335073b0"
server_key_iv: "c559e491833f06909f8b72bdae5aeaac"
server_key_version: "encryptionkey@0"
I20260809 03:32:43.945091 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 03:32:43.947383 5576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:43.948522 5557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:32:43.948644 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a"
format_stamp: "Formatted at 2026-08-09 03:32:43 on dist-test-slave-cvwc"
server_key: "6f6602bf45e67b76a396c7ff335073b0"
server_key_iv: "c559e491833f06909f8b72bdae5aeaac"
server_key_version: "encryptionkey@0"
I20260809 03:32:43.948734 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:43.963222 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:43.965790 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:43.965961 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:43.972582 5557 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46203
I20260809 03:32:43.972841 5657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46203 every 8 connection(s)
I20260809 03:32:43.973517 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:43.976218 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:43.979327 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5557
I20260809 03:32:43.979449 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:32:43.979700 16249 external_mini_cluster.cc:1428] Setting key 454c28956fcc515c89bcedd5197a599a
I20260809 03:32:43.981956 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap starting.
I20260809 03:32:43.984409 5658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:43.985260 5658 log.cc:824] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:43.987427 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: No bootstrap required, opened a new log
Aug 09 03:32:43 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:43.991381 5658 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:43.991667 5658 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:43.991734 5658 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Initialized, Role: FOLLOWER
I20260809 03:32:43.992185 5658 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:43.992349 5658 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:43.992491 5658 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:43.992607 5658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:43.993475 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:43.980968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.15.222.126:46203)
0809 03:32:43.981353 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:43.981368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:43.981412 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:32:43.982042 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:43.982063 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:43.982422 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:43.982692 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:43.983570 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:43.984549 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:43.985674 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:43.986284 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:43.986405 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:43.989083 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:43.989088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:43.989101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:43.989126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:43.990715 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:43.991183 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:43.991188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:43.991192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:43.991292 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:43.991557 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:43.991559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:43.991560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:43.991992 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:43.992157 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:43.992471 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 03:32:43.992762 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260809 03:32:43.993471 5658 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:43.993873 5658 leader_election.cc:302] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b741e56b8b4ce3b8968ee5faa4af9a; no voters:
I20260809 03:32:43.994159 5658 leader_election.cc:288] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:32:43.994246 5666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:43.994412 5666 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 LEADER]: Becoming Leader. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Running, Role: LEADER
I20260809 03:32:43.995373 5658 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:43.995819 5666 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:43.997375 5668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b741e56b8b4ce3b8968ee5faa4af9a. Latest consensus state: current_term: 1 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:43.997695 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:43.997824 5668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:43.997823 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:43.998220 5689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:44.000706 5689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:44.005940 5689 catalog_manager.cc:1347] Generated new cluster ID: 4eda91d7033941cdbf47bdbfd464a207
I20260809 03:32:44.006000 5689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:44.020839 5689 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:32:44.022197 5689 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:44.027498 5689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Generated new TSK 0
I20260809 03:32:44.028117 5689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:32:44.062500 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:44.196211 5697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:44.196447 5697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:44.196494 5697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:44.200410 5697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:44.200464 5697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:44.200531 5697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:32:44.205471 5697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:44.206384 5697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:44.207322 5697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:44.216148 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.216219 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.216147 5703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:44.216308 5697 server_base.cc:1034] running on GCE node
I20260809 03:32:44.216763 5697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:44.217329 5697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:44.218495 5697 hybrid_clock.cc:648] HybridClock initialized: now 1786246364218457 us; error 38 us; skew 500 ppm
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:44.221055 5697 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:32:44.221836 5697 webserver.cc:460] Webserver started at http://127.15.222.65:41415/ using document root <none> and password file <none>
I20260809 03:32:44.222335 5697 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:44.222398 5697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:44.222569 5697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:44.224449 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "56eaecdb9ef547fc9af17e9f4d331388"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "769d2cd44e4d529c9610e4b959d6d16f"
server_key_iv: "47df86e659c23be56bf2d0f2fc395d0e"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.224967 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "56eaecdb9ef547fc9af17e9f4d331388"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "769d2cd44e4d529c9610e4b959d6d16f"
server_key_iv: "47df86e659c23be56bf2d0f2fc395d0e"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.228307 5697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:32:44.230484 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.231559 5697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:44.231669 5697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "56eaecdb9ef547fc9af17e9f4d331388"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "769d2cd44e4d529c9610e4b959d6d16f"
server_key_iv: "47df86e659c23be56bf2d0f2fc395d0e"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.231753 5697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:44.241384 5697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:44.243763 5697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:44.243924 5697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:44.244423 5697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:44.245153 5697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:44.245199 5697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.245246 5697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:44.245275 5697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.253194 5697 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:45759
I20260809 03:32:44.253238 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:45759 every 8 connection(s)
I20260809 03:32:44.254114 5697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:32:44.258272 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5697
I20260809 03:32:44.258406 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:32:44.258731 16249 external_mini_cluster.cc:1428] Setting key 5cb706fe646778b6bc3ace9373fcfb45
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:44.267375 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.255715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:45727 (local address 127.15.222.126:46203)
0809 03:32:44.255866 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.255870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.256615 (+ 745us) server_negotiation.cc:407] Connection header received
0809 03:32:44.257906 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.257908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.257954 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.258028 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:44.259397 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.259991 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.261069 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.261294 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.261379 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.263939 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:44.263943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:44.263945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:44.263972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:44.265328 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.265930 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.265933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.265934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.266004 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.266346 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.266348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.266349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.266512 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:44.266580 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.267152 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.267245 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:32:44.268445 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.256066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:45727)
0809 03:32:44.256477 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.256517 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.257698 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.258219 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.258229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.258714 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:44.259275 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.259290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.260105 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.260109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.260956 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.260966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.261440 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.261444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.261594 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.262152 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:44.262171 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:44.263770 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:44.265441 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.265448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.265463 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.265838 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.266085 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.266088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.266089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.266266 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.266605 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:44.266611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:44.266929 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.267124 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.267474 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:32:44.269634 5899 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:44.269872 5899 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:44.270378 5899 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:32:44.271870 5601 ts_manager.cc:194] Registered new tserver with Master: 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:44.273007 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:45727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:32:44.286048 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:44.393049 5904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:44.393355 5904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:44.393414 5904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:44.397585 5904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:44.397650 5904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:44.397770 5904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:32:44.402892 5904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:44.403854 5904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:44.404835 5904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:44.414037 5915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.414028 5913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.414137 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:44.414268 5904 server_base.cc:1034] running on GCE node
I20260809 03:32:44.414608 5904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:44.415212 5904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:44.416389 5904 hybrid_clock.cc:648] HybridClock initialized: now 1786246364416356 us; error 39 us; skew 500 ppm
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:44.419163 5904 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:32:44.419952 5904 webserver.cc:460] Webserver started at http://127.15.222.66:36729/ using document root <none> and password file <none>
I20260809 03:32:44.420506 5904 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:44.420568 5904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:44.420738 5904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:44.422668 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "8727f41110be460c9f1e0958fce0609c"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "a588cc46a81d404460be83ddd54b0e15"
server_key_iv: "a610b789f1e178f011702ef2baaf7196"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.423161 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "8727f41110be460c9f1e0958fce0609c"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "a588cc46a81d404460be83ddd54b0e15"
server_key_iv: "a610b789f1e178f011702ef2baaf7196"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.426517 5904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:32:44.428620 5928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.429731 5904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:44.429839 5904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "8727f41110be460c9f1e0958fce0609c"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "a588cc46a81d404460be83ddd54b0e15"
server_key_iv: "a610b789f1e178f011702ef2baaf7196"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.429924 5904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:44.443154 5904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:44.445640 5904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:44.445830 5904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:44.446375 5904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:44.447178 5904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:44.447225 5904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.447266 5904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:44.447295 5904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.455166 5904 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42071
I20260809 03:32:44.455214 6098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42071 every 8 connection(s)
I20260809 03:32:44.455978 5904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:32:44.461354 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 5904
I20260809 03:32:44.461486 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:32:44.461777 16249 external_mini_cluster.cc:1428] Setting key 8fa2e66c82376a6e4a94a9f7ff61243f
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:44.468791 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.457548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:47669 (local address 127.15.222.126:46203)
0809 03:32:44.457659 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.457662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.458467 (+ 805us) server_negotiation.cc:407] Connection header received
0809 03:32:44.459738 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.459740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.459787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.459857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:44.460869 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.461401 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.462749 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.463016 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.463105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.465339 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:44.465342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:44.465345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:44.465372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:44.466805 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.467252 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.467255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.467257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.467344 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.467620 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.467623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.467625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.467846 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:44.467970 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.468498 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.468626 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:32:44.469506 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.457907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:47669)
0809 03:32:44.458321 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.458360 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.459564 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.459997 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.460004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.460320 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:44.460763 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.460775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.461522 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.461524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.462638 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.462649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.463157 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.463162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.463321 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.463819 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:44.463834 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:44.465180 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:44.466908 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.466912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.466923 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.467161 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.467424 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.467426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.467427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.467545 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.467951 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:44.467957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:44.468251 (+ 294us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.468430 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.468664 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260809 03:32:44.470592 6101 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:44.470837 6101 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:44.471390 6101 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:32:44.472462 5601 ts_manager.cc:194] Registered new tserver with Master: 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
I20260809 03:32:44.472998 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:32:44.488386 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:44.598341 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:44.598613 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:44.598654 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:44.602550 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:44.602595 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:44.602650 6107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:32:44.607700 6107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:44.608630 6107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:44.609565 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:44.619174 6114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.619179 6117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:44.619210 6115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:44.619365 6107 server_base.cc:1034] running on GCE node
I20260809 03:32:44.619767 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:44.620455 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:44.621621 6107 hybrid_clock.cc:648] HybridClock initialized: now 1786246364621600 us; error 34 us; skew 500 ppm
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:44.624219 6107 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:32:44.625033 6107 webserver.cc:460] Webserver started at http://127.15.222.67:38295/ using document root <none> and password file <none>
I20260809 03:32:44.625563 6107 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:44.625630 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:44.625857 6107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:32:44.628058 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "d9a320d79a8841628143e55c65dda5f7"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "fcf383817ab0d1f18f093ab6a3f336f3"
server_key_iv: "376cab2cf49a811f3e550a13c46587f9"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.628614 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "d9a320d79a8841628143e55c65dda5f7"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "fcf383817ab0d1f18f093ab6a3f336f3"
server_key_iv: "376cab2cf49a811f3e550a13c46587f9"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.632213 6107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:32:44.634641 6126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.635856 6107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:44.635968 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d9a320d79a8841628143e55c65dda5f7"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "fcf383817ab0d1f18f093ab6a3f336f3"
server_key_iv: "376cab2cf49a811f3e550a13c46587f9"
server_key_version: "encryptionkey@0"
I20260809 03:32:44.636058 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:44.646864 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:44.649541 6107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:44.649734 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:44.650286 6107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:44.651065 6107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:44.651113 6107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.651165 6107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:44.651193 6107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:44.659541 6107 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34345
I20260809 03:32:44.659564 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34345 every 8 connection(s)
I20260809 03:32:44.660377 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:32:44.664489 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 6107
I20260809 03:32:44.664605 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:32:44.664862 16249 external_mini_cluster.cc:1428] Setting key d6d9a9ab509afbdba523109c89d91cd9
Aug 09 03:32:44 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:44.673800 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.662089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:58973 (local address 127.15.222.126:46203)
0809 03:32:44.662223 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.662227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.662989 (+ 762us) server_negotiation.cc:407] Connection header received
0809 03:32:44.664291 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.664295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.664349 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.664452 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:44.665819 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.666562 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.668104 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.668367 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.668474 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.670801 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:44.670805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:44.670807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:44.670833 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:44.671931 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.672373 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.672376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.672378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.672450 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.672747 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.672750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.672752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.672940 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:44.673025 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.673496 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.673614 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:32:44.674546 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.662420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:58973)
0809 03:32:44.662840 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.662880 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.664096 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.664601 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.664610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.665021 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:44.665653 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.665673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.666731 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.666735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.667992 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.668003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.668523 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.668527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.668698 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.669222 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:44.669238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:44.670649 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:44.672032 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.672037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.672051 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.672283 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.672527 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:44.672530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:44.672532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:44.672663 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:44.673035 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:44.673041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:44.673269 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.673438 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.673771 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260809 03:32:44.675513 6298 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:44.675714 6298 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:44.676143 6298 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
I20260809 03:32:44.677055 5601 ts_manager.cc:194] Registered new tserver with Master: d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345)
I20260809 03:32:44.677651 5601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:58973
I20260809 03:32:44.678977 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:32:44.686932 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.680821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.15.222.126:46203)
0809 03:32:44.680940 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.680943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.681087 (+ 144us) server_negotiation.cc:407] Connection header received
0809 03:32:44.681181 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.681183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.681217 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.681277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:44.681905 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.682381 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.683108 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.683300 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.683362 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.684505 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:44.684509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:44.684511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:44.684532 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:44.685600 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.686004 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.686006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.686007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.686074 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:44.686265 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:44.686266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:44.686267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:44.686465 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:44.686563 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.686712 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.686800 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:32:44.689509 5601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:32:44.692521 5601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:32:44.706522 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.701003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.0.0.1:53990)
0809 03:32:44.701895 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.701908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.702078 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.702620 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.702625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.702648 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:44.702773 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.702780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.703818 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.703822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.704982 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.704993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.706003 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.706006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.706112 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706148 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.706249 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.706364 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":84,"threads_started":1}
I20260809 03:32:44.706529 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.701181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.15.222.65:45759)
0809 03:32:44.701588 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.701600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.702003 (+ 403us) server_negotiation.cc:407] Connection header received
0809 03:32:44.702243 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.702248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.702453 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.702599 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:44.702905 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.703696 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.705120 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.705899 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.705994 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706142 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.706288 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.706380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":137,"threads_started":1}
I20260809 03:32:44.706950 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.700758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42071 (local address 127.0.0.1:52186)
0809 03:32:44.701695 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.701766 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.701966 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.702719 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.702723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.702741 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:44.702845 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.702852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.704311 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.704314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.705335 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.705344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.706566 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.706570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.706657 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706680 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.706752 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.706815 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":41,"thread_start_us":117,"threads_started":1}
I20260809 03:32:44.707039 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.700924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.15.222.66:42071)
0809 03:32:44.701380 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.701407 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.701855 (+ 448us) server_negotiation.cc:407] Connection header received
0809 03:32:44.702159 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.702166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.702374 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.702516 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:44.702982 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.704177 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.705448 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.706250 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.706351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706504 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.706791 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.706887 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":138,"threads_started":1}
I20260809 03:32:44.707078 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.701497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34345 (local address 127.0.0.1:45530)
0809 03:32:44.702316 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.702328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.702417 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.702791 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.702794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.702810 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:44.702912 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.702919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.704288 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.704291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.705472 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.705483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.706618 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.706621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.706822 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706843 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.706914 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.706969 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260809 03:32:44.707448 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.701875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.15.222.67:34345)
0809 03:32:44.702305 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.702314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.702387 (+ 73us) server_negotiation.cc:407] Connection header received
0809 03:32:44.702515 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.702519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.702679 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.702812 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:44.703029 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.704145 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.705585 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.706370 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.706460 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.706569 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.707213 (+ 644us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.707305 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":141,"threads_started":1}
I20260809 03:32:44.710143 5803 tablet_service.cc:1467] Processing CreateTablet for tablet 35ea30d619bd45779d4df9b267358911 (DEFAULT_TABLE table=test-table [id=1528e75ec03f4b23b516b2d9617e4e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:44.710054 5994 tablet_service.cc:1467] Processing CreateTablet for tablet 35ea30d619bd45779d4df9b267358911 (DEFAULT_TABLE table=test-table [id=1528e75ec03f4b23b516b2d9617e4e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:44.710054 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 35ea30d619bd45779d4df9b267358911 (DEFAULT_TABLE table=test-table [id=1528e75ec03f4b23b516b2d9617e4e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:44.711057 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ea30d619bd45779d4df9b267358911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:44.711084 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ea30d619bd45779d4df9b267358911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:44.711089 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ea30d619bd45779d4df9b267358911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:44.717351 6323 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap starting.
I20260809 03:32:44.717799 6324 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Bootstrap starting.
I20260809 03:32:44.720293 6326 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Bootstrap starting.
I20260809 03:32:44.720314 6323 tablet_bootstrap.cc:654] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:44.720314 6324 tablet_bootstrap.cc:654] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:44.721190 6324 log.cc:824] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:44.721190 6323 log.cc:824] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:44.722592 6326 tablet_bootstrap.cc:654] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:44.723253 6326 log.cc:824] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:44.723307 6324 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: No bootstrap required, opened a new log
I20260809 03:32:44.723307 6323 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: No bootstrap required, opened a new log
I20260809 03:32:44.723484 6323 ts_tablet_manager.cc:1397] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:32:44.723484 6324 ts_tablet_manager.cc:1397] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:32:44.724764 6326 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: No bootstrap required, opened a new log
I20260809 03:32:44.724941 6326 ts_tablet_manager.cc:1397] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:32:44.727216 6324 raft_consensus.cc:348] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.727216 6323 raft_consensus.cc:348] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.727474 6323 raft_consensus.cc:374] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:44.727474 6324 raft_consensus.cc:374] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:44.727541 6324 raft_consensus.cc:729] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a320d79a8841628143e55c65dda5f7, State: Initialized, Role: FOLLOWER
I20260809 03:32:44.727560 6323 raft_consensus.cc:729] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaecdb9ef547fc9af17e9f4d331388, State: Initialized, Role: FOLLOWER
I20260809 03:32:44.727588 6326 raft_consensus.cc:348] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.727857 6326 raft_consensus.cc:374] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:44.727921 6326 raft_consensus.cc:729] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8727f41110be460c9f1e0958fce0609c, State: Initialized, Role: FOLLOWER
I20260809 03:32:44.728046 6324 consensus_queue.cc:260] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.728046 6323 consensus_queue.cc:260] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.728312 6326 consensus_queue.cc:260] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.728945 6298 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
I20260809 03:32:44.729036 6323 ts_tablet_manager.cc:1428] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:32:44.729266 6326 ts_tablet_manager.cc:1428] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 03:32:44.729589 6101 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
I20260809 03:32:44.729782 6324 ts_tablet_manager.cc:1428] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:32:44.730166 5899 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
W20260809 03:32:44.755111 5900 tablet.cc:2369] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:44.901911 6336 raft_consensus.cc:482] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:44.902148 6336 raft_consensus.cc:504] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.903331 6336 leader_election.cc:288] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:44.906975 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.903582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.15.222.66:49889)
0809 03:32:44.903687 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.903702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.903797 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.904084 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.904086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.904106 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:44.904229 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.904236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.905031 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.905034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.905922 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.905932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.906580 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.906682 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.906708 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.906775 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.906836 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 03:32:44.906992 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.903702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:49889 (local address 127.15.222.65:45759)
0809 03:32:44.903825 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.903830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.903845 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:32:44.903905 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.903908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.903962 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.904051 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:44.904341 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.904928 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906046 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.906470 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906559 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.906599 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.906804 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.906873 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:32:44.907121 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.903718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40759 (local address 127.15.222.67:34345)
0809 03:32:44.903830 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.903834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.904170 (+ 336us) server_negotiation.cc:407] Connection header received
0809 03:32:44.904267 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.904270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.904322 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.904384 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:44.904643 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.905165 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906186 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.906605 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.906761 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.906936 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.907016 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:32:44.907133 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.903582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34345 (local address 127.15.222.66:40759)
0809 03:32:44.904063 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.904076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:44.904175 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:44.904413 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:44.904416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:44.904434 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:44.904543 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.904549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.905271 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.905273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.906068 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:44.906078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.906717 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.906720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:44.906814 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.906836 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:32:44.906917 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:32:44.906979 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":110,"thread_start_us":144,"threads_started":1}
I20260809 03:32:44.907807 5834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ea30d619bd45779d4df9b267358911" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaecdb9ef547fc9af17e9f4d331388" is_pre_election: true
I20260809 03:32:44.907799 6233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ea30d619bd45779d4df9b267358911" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a320d79a8841628143e55c65dda5f7" is_pre_election: true
I20260809 03:32:44.908079 6233 raft_consensus.cc:2454] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 0.
I20260809 03:32:44.908093 5834 raft_consensus.cc:2454] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 0.
I20260809 03:32:44.908504 5933 leader_election.cc:302] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388, 8727f41110be460c9f1e0958fce0609c; no voters:
I20260809 03:32:44.908759 6336 raft_consensus.cc:2781] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:32:44.908845 6336 raft_consensus.cc:482] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:32:44.908901 6336 raft_consensus.cc:3037] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:44.909929 6336 raft_consensus.cc:504] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.910216 6336 leader_election.cc:288] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 election: Requested vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:44.910534 6233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ea30d619bd45779d4df9b267358911" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a320d79a8841628143e55c65dda5f7"
I20260809 03:32:44.910580 5834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ea30d619bd45779d4df9b267358911" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaecdb9ef547fc9af17e9f4d331388"
I20260809 03:32:44.910660 6233 raft_consensus.cc:3037] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:44.910686 5834 raft_consensus.cc:3037] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Advancing to term 1
W20260809 03:32:44.911288 6300 tablet.cc:2369] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:44.911693 6233 raft_consensus.cc:2454] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 1.
I20260809 03:32:44.911846 5834 raft_consensus.cc:2454] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 1.
I20260809 03:32:44.911969 5929 leader_election.cc:302] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7; no voters:
I20260809 03:32:44.912139 6336 raft_consensus.cc:2781] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:44.912352 6336 raft_consensus.cc:686] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 LEADER]: Becoming Leader. State: Replica: 8727f41110be460c9f1e0958fce0609c, State: Running, Role: LEADER
I20260809 03:32:44.912676 6336 consensus_queue.cc:237] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:44.915685 5600 catalog_manager.cc:5495] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8727f41110be460c9f1e0958fce0609c (127.15.222.66). New cstate: current_term: 1 leader_uuid: "8727f41110be460c9f1e0958fce0609c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } health_report { overall_health: UNKNOWN } } }
I20260809 03:32:44.937361 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.933991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.15.222.66:42071)
0809 03:32:44.934104 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:44.934109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:44.934143 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:32:44.934290 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:44.934292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:44.934346 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:44.934439 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:44.934697 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.935203 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.936017 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:44.936226 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:44.936310 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:44.936595 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:44.936990 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:44.937094 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:44.937162 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 03:32:44.937217 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:32:44.943216 5834 raft_consensus.cc:1261] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Refusing update from remote peer 8727f41110be460c9f1e0958fce0609c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:44.943285 6233 raft_consensus.cc:1261] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Refusing update from remote peer 8727f41110be460c9f1e0958fce0609c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:44.943913 6338 consensus_queue.cc:1035] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:44.944072 6336 consensus_queue.cc:1035] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:44.958675 5598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:44.958844 5598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:44.961091 5598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Sending DeleteTablet for 3 replicas of tablet 35ea30d619bd45779d4df9b267358911
I20260809 03:32:44.961874 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ea30d619bd45779d4df9b267358911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:44 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260809 03:32:44.961872 5803 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ea30d619bd45779d4df9b267358911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:44 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:53990
I20260809 03:32:44.962040 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5697
I20260809 03:32:44.977942 5994 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ea30d619bd45779d4df9b267358911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:44 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52186
I20260809 03:32:44.978420 6360 tablet_replica.cc:330] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: stopping tablet replica
I20260809 03:32:44.978818 6360 raft_consensus.cc:2229] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:32:44.979305 6360 raft_consensus.cc:2258] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:44.981107 6360 ts_tablet_manager.cc:1905] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:32:44.982800 5581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:45759 (error 108)
I20260809 03:32:44.983741 6361 tablet_replica.cc:330] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: stopping tablet replica
I20260809 03:32:44.983959 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5904
I20260809 03:32:44.986121 6360 ts_tablet_manager.cc:1918] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:44.986218 6360 log.cc:1196] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/35ea30d619bd45779d4df9b267358911
I20260809 03:32:44.986485 6360 ts_tablet_manager.cc:1939] T 35ea30d619bd45779d4df9b267358911 P d9a320d79a8841628143e55c65dda5f7: Deleting consensus metadata
I20260809 03:32:44.987601 5578 catalog_manager.cc:4841] TS d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): tablet 35ea30d619bd45779d4df9b267358911 (table test-table [id=1528e75ec03f4b23b516b2d9617e4e76]) successfully deleted
I20260809 03:32:44.990384 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.990122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.0.0.1:53996)
0809 03:32:44.990250 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.990317 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:45759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 03:32:44.990566 5581 catalog_manager.cc:4584] TS 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35ea30d619bd45779d4df9b267358911: Network error: Client connection negotiation failed: client connection to 127.15.222.65:45759: connect: Connection refused (error 111)
I20260809 03:32:44.995080 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 6107
I20260809 03:32:44.998725 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:44.995167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42071 (local address 127.0.0.1:52214)
0809 03:32:44.998599 (+ 3432us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:44.998656 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.66:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3347}
I20260809 03:32:45.008074 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 5557
I20260809 03:32:45.018800 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--webserver_interface=127.15.222.126
--webserver_port=42903
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:45.127720 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:45.127947 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:45.127988 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:45.131794 6364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:45.131839 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:45.131860 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:45.131894 6364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:45.131922 6364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:45.136898 6364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:45.137789 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:45.138732 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:32:45.151732 6364 server_base.cc:1034] running on GCE node
W20260809 03:32:45.151631 6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.151662 6376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.151631 6379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:45.152539 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:45.153604 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:45.154759 6364 hybrid_clock.cc:648] HybridClock initialized: now 1786246365154736 us; error 34 us; skew 500 ppm
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:45.157311 6364 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:45.158210 6364 webserver.cc:460] Webserver started at http://127.15.222.126:42903/ using document root <none> and password file <none>
I20260809 03:32:45.158783 6364 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:45.158869 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:45.162542 6364 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:45.164520 6387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.165649 6364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:45.165786 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a"
format_stamp: "Formatted at 2026-08-09 03:32:43 on dist-test-slave-cvwc"
server_key: "6f6602bf45e67b76a396c7ff335073b0"
server_key_iv: "c559e491833f06909f8b72bdae5aeaac"
server_key_version: "encryptionkey@0"
I20260809 03:32:45.166347 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:45.181517 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:45.184150 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:45.184321 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:45.191170 6364 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46203
I20260809 03:32:45.191489 6467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46203 every 8 connection(s)
I20260809 03:32:45.192284 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:45.194931 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 6364
I20260809 03:32:45.195322 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:45759
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=41415
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:45.196626 6471 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:32:45.198726 6471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap starting.
I20260809 03:32:45.214980 6471 log.cc:824] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:45.223352 6471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:45.223656 6471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap complete.
I20260809 03:32:45.227412 6471 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:45.227777 6471 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.228250 6471 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:45.228375 6471 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:45.228447 6471 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:45.228543 6471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:32:45.230334 6471 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:45.230638 6471 leader_election.cc:302] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b741e56b8b4ce3b8968ee5faa4af9a; no voters:
I20260809 03:32:45.230906 6471 leader_election.cc:288] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:32:45.231043 6477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:32:45.231266 6477 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 LEADER]: Becoming Leader. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Running, Role: LEADER
I20260809 03:32:45.231616 6477 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:45.232017 6471 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:45.232554 6477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b741e56b8b4ce3b8968ee5faa4af9a. Latest consensus state: current_term: 2 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:45.232669 6477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:45.233063 6478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:45.233165 6478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:45.233448 6488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:45.237978 6488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:45.238636 6488 catalog_manager.cc:1259] Loaded cluster ID: 4eda91d7033941cdbf47bdbfd464a207
I20260809 03:32:45.238693 6488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:32:45.240082 6488 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 03:32:45.244513 6488 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 03:32:45.250079 6488 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:32:45.250934 6488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Loaded TSK: 0
I20260809 03:32:45.252229 6488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:32:45.313748 6474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:45.313997 6474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:45.314040 6474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:45.317948 6474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:45.318002 6474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:45.318066 6474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:32:45.323212 6474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:45759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=41415
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:45.324172 6474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:45.325140 6474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:45.335641 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.335676 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.335668 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:45.336692 6474 server_base.cc:1034] running on GCE node
I20260809 03:32:45.337100 6474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:45.337775 6474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:45.338954 6474 hybrid_clock.cc:648] HybridClock initialized: now 1786246365338924 us; error 40 us; skew 500 ppm
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:45.341902 6474 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:32:45.342819 6474 webserver.cc:460] Webserver started at http://127.15.222.65:41415/ using document root <none> and password file <none>
I20260809 03:32:45.343387 6474 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:45.343473 6474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:45.347249 6474 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 03:32:45.349244 6544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.350457 6474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:45.350549 6474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "56eaecdb9ef547fc9af17e9f4d331388"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "769d2cd44e4d529c9610e4b959d6d16f"
server_key_iv: "47df86e659c23be56bf2d0f2fc395d0e"
server_key_version: "encryptionkey@0"
I20260809 03:32:45.351069 6474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:45.367700 6474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:45.370914 6474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:45.371095 6474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:45.371713 6474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:45.373008 6554 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:32:45.377398 6474 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:32:45.377476 6474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260809 03:32:45.377554 6474 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:32:45.379319 6474 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:32:45.379365 6474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 03:32:45.379706 6554 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap starting.
I20260809 03:32:45.388793 6474 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:45759
I20260809 03:32:45.388937 6719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:45759 every 8 connection(s)
I20260809 03:32:45.389796 6474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:32:45.393110 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 6474
I20260809 03:32:45.393675 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:42071
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=36729
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:45.399626 6554 log.cc:824] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:45.407681 6554 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:45.408120 6554 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap complete.
I20260809 03:32:45.408612 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.391787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:43785)
0809 03:32:45.392246 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.392298 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.393122 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.394370 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.394378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.394782 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:45.395232 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.395257 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.397056 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.397060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.398514 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.398527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.399578 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.399583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.399872 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.400583 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:45.400605 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:45.402552 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:45.405123 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.405130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.405144 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.405515 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.405964 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.405967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.405969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.406147 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.406817 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:45.406824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:45.407107 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.407332 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.407658 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260809 03:32:45.409161 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.391861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:43785 (local address 127.15.222.126:46203)
0809 03:32:45.392396 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.392417 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.392469 (+ 52us) server_negotiation.cc:407] Connection header received
0809 03:32:45.393588 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.393599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.393995 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.394290 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:45.395404 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.396849 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.398704 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.399413 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.399600 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.402769 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:45.402776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:45.402794 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:45.402829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:45.404991 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.405656 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.405663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.405670 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.405857 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.406268 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.406271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.406273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.406697 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:45.406825 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.407442 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.407730 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":128,"threads_started":1}
I20260809 03:32:45.409372 6554 ts_tablet_manager.cc:1397] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20260809 03:32:45.411509 6721 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:45.411747 6721 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:45.412307 6721 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
I20260809 03:32:45.412142 6554 raft_consensus.cc:348] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.412597 6554 raft_consensus.cc:729] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaecdb9ef547fc9af17e9f4d331388, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.413074 6554 consensus_queue.cc:260] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.414047 6554 ts_tablet_manager.cc:1428] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260809 03:32:45.415673 6412 ts_manager.cc:194] Registered new tserver with Master: 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:45.418959 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:43785
I20260809 03:32:45.421371 6721 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:32:45.428692 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.418313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.0.0.1:54012)
0809 03:32:45.419047 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.419093 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.419230 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.419847 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.419853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.420269 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:45.420397 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.420405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.421700 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.421704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.422641 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.422653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.423317 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.423320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.423430 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.423929 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:45.423943 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:45.425350 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:45.427067 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.427075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.427081 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.427310 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.427584 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.427588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.427590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.427671 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.428099 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:45.428104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:45.428216 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.428340 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.428524 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":111,"threads_started":1}
I20260809 03:32:45.428711 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.418406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.15.222.65:45759)
0809 03:32:45.418774 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.418785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.419180 (+ 395us) server_negotiation.cc:407] Connection header received
0809 03:32:45.419386 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.419391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.419584 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.419730 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:45.420523 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.421575 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.422752 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.423191 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.423274 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.425476 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:45.425486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:45.425499 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:45.425520 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:45.426954 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.427415 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.427420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.427421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.427500 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.427750 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.427754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.427755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.427991 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:45.428104 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.428432 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.428577 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260809 03:32:45.429801 6619 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ea30d619bd45779d4df9b267358911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:44 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:54012
I20260809 03:32:45.430586 6731 tablet_replica.cc:330] stopping tablet replica
I20260809 03:32:45.430857 6731 raft_consensus.cc:2229] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:32:45.431075 6731 raft_consensus.cc:2258] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:45.431998 6731 ts_tablet_manager.cc:1905] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:45.435576 6731 ts_tablet_manager.cc:1918] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:45.435662 6731 log.cc:1196] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/35ea30d619bd45779d4df9b267358911
I20260809 03:32:45.435921 6731 ts_tablet_manager.cc:1939] T 35ea30d619bd45779d4df9b267358911 P 56eaecdb9ef547fc9af17e9f4d331388: Deleting consensus metadata
I20260809 03:32:45.437440 6390 catalog_manager.cc:4841] TS 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759): tablet 35ea30d619bd45779d4df9b267358911 (table test-table [id=1528e75ec03f4b23b516b2d9617e4e76]) successfully deleted
W20260809 03:32:45.511340 6725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:45.511579 6725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:45.511634 6725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:45.515635 6725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:45.515692 6725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:45.515774 6725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:32:45.520895 6725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:42071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=36729
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:45.521910 6725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:45.522914 6725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:45.532063 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.532058 6746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.532058 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:45.532425 6725 server_base.cc:1034] running on GCE node
I20260809 03:32:45.532897 6725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:45.533490 6725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:45.534642 6725 hybrid_clock.cc:648] HybridClock initialized: now 1786246365534617 us; error 34 us; skew 500 ppm
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:45.537171 6725 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:32:45.537990 6725 webserver.cc:460] Webserver started at http://127.15.222.66:36729/ using document root <none> and password file <none>
I20260809 03:32:45.538502 6725 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:45.538573 6725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:45.542194 6725 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:45.544315 6756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.545449 6725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:45.545544 6725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "8727f41110be460c9f1e0958fce0609c"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "a588cc46a81d404460be83ddd54b0e15"
server_key_iv: "a610b789f1e178f011702ef2baaf7196"
server_key_version: "encryptionkey@0"
I20260809 03:32:45.546103 6725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:45.557294 6725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:45.560003 6725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:45.560140 6725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:45.560626 6725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:45.561610 6766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:32:45.565663 6725 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:32:45.565783 6725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 03:32:45.565845 6725 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:32:45.567456 6725 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:32:45.567497 6725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 03:32:45.567751 6766 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Bootstrap starting.
I20260809 03:32:45.577268 6725 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42071
I20260809 03:32:45.577323 6913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42071 every 8 connection(s)
I20260809 03:32:45.578189 6725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:32:45.581457 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 6725
I20260809 03:32:45.581955 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:34345
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=38295
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:45.586966 6766 log.cc:824] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:45.591776 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.580062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:39995 (local address 127.15.222.126:46203)
0809 03:32:45.580197 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.580201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.580973 (+ 772us) server_negotiation.cc:407] Connection header received
0809 03:32:45.582311 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.582315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.582387 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.582494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:45.583975 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.584507 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.585611 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.585976 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.586084 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.588522 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:45.588525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:45.588527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:45.588552 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:45.589900 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.590361 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.590364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.590366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.590446 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.590745 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.590749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.590751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.590933 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:45.591018 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.591479 (+ 461us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.591613 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:32:45.592687 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.580289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:39995)
0809 03:32:45.580826 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.580869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.581854 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.582684 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.582694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.583139 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:45.583787 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.583809 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.584635 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.584637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.585488 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.585497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.586101 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.586104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.586240 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.586792 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:45.586808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:45.588360 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:45.590008 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.590013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.590027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.590270 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.590527 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.590530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.590532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.590660 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.591029 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:45.591035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:45.591260 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.591416 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.591714 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260809 03:32:45.593768 6916 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:45.594000 6916 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:45.594450 6916 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
I20260809 03:32:45.596042 6412 ts_manager.cc:194] Registered new tserver with Master: 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
I20260809 03:32:45.596086 6766 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:45.596467 6766 tablet_bootstrap.cc:492] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Bootstrap complete.
I20260809 03:32:45.597190 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:39995
I20260809 03:32:45.597904 6766 ts_tablet_manager.cc:1397] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Time spent bootstrapping tablet: real 0.030s user 0.029s sys 0.000s
I20260809 03:32:45.600800 6766 raft_consensus.cc:348] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.601320 6766 raft_consensus.cc:729] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8727f41110be460c9f1e0958fce0609c, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.601883 6766 consensus_queue.cc:260] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.602006 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.596907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42071 (local address 127.0.0.1:52230)
0809 03:32:45.597486 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.597500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.597576 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.598761 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.598765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.598784 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:45.598873 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.598879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.599990 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.599993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.600773 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.600781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.601537 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.601541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.601637 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.601693 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.601796 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.601881 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514}
I20260809 03:32:45.602013 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.597746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.15.222.66:42071)
0809 03:32:45.598110 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.598121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.598142 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:32:45.598203 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.598207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.598590 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.599012 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:45.599107 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.599885 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.600884 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.601428 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.601525 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.601652 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.601826 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.601908 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":103,"threads_started":1}
I20260809 03:32:45.602921 6766 ts_tablet_manager.cc:1428] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:32:45.603048 6916 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
I20260809 03:32:45.602991 6819 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ea30d619bd45779d4df9b267358911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:44 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:32:45.603852 6929 tablet_replica.cc:330] stopping tablet replica
I20260809 03:32:45.604139 6929 raft_consensus.cc:2229] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:32:45.604370 6929 raft_consensus.cc:2258] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:45.605386 6929 ts_tablet_manager.cc:1905] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:45.609890 6929 ts_tablet_manager.cc:1918] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:45.609993 6929 log.cc:1196] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/35ea30d619bd45779d4df9b267358911
I20260809 03:32:45.610252 6929 ts_tablet_manager.cc:1939] T 35ea30d619bd45779d4df9b267358911 P 8727f41110be460c9f1e0958fce0609c: Deleting consensus metadata
I20260809 03:32:45.612071 6388 catalog_manager.cc:4841] TS 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): tablet 35ea30d619bd45779d4df9b267358911 (table test-table [id=1528e75ec03f4b23b516b2d9617e4e76]) successfully deleted
W20260809 03:32:45.703799 6921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:45.704048 6921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:45.704094 6921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:45.708050 6921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:45.708105 6921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:45.708176 6921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:32:45.713397 6921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:34345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=38295
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:45.714454 6921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:45.715474 6921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:45.724823 6934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:45.725071 6921 server_base.cc:1034] running on GCE node
W20260809 03:32:45.724831 6938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:45.724841 6935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:45.725621 6921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:45.726243 6921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:45.727399 6921 hybrid_clock.cc:648] HybridClock initialized: now 1786246365727381 us; error 31 us; skew 500 ppm
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:45.730067 6921 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:32:45.730890 6921 webserver.cc:460] Webserver started at http://127.15.222.67:38295/ using document root <none> and password file <none>
I20260809 03:32:45.731494 6921 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:45.731585 6921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:45.735685 6921 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:32:45.737774 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.739015 6921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:45.739125 6921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d9a320d79a8841628143e55c65dda5f7"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "fcf383817ab0d1f18f093ab6a3f336f3"
server_key_iv: "376cab2cf49a811f3e550a13c46587f9"
server_key_version: "encryptionkey@0"
I20260809 03:32:45.739660 6921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:45.750227 6921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:45.752803 6921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:45.752969 6921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:45.753515 6921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:45.754372 6921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:45.754420 6921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.754470 6921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:45.754496 6921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:45.763072 6921 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34345
I20260809 03:32:45.763094 7119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34345 every 8 connection(s)
I20260809 03:32:45.764034 6921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:32:45.770426 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 6921
Aug 09 03:32:45 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:32:45.778046 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.765819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:57683 (local address 127.15.222.126:46203)
0809 03:32:45.765948 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.765951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.766751 (+ 800us) server_negotiation.cc:407] Connection header received
0809 03:32:45.768122 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.768126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.768185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.768255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:45.769395 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.770009 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.771471 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.771809 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.771919 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.774440 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:45.774443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:45.774447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:45.774474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:45.775959 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.776512 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.776516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.776518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.776609 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.776966 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.776970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.776972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.777155 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:45.777252 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.777818 (+ 566us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.777920 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:32:45.778967 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.766171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:57683)
0809 03:32:45.766615 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.766655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.767919 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.768449 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.768456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.768800 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:45.769255 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.769268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.770132 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.770135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.771330 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.771340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.771962 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.771966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.772135 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.772684 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:45.772705 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:45.774278 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:45.776078 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.776084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.776099 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.776410 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.776723 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:45.776727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:45.776729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:45.776870 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:45.777270 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:45.777277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:45.777538 (+ 261us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.777775 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.778121 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260809 03:32:45.779340 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.771113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.15.222.126:46203)
0809 03:32:45.771368 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.771372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.771635 (+ 263us) server_negotiation.cc:407] Connection header received
0809 03:32:45.771780 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.771782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.771822 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.771910 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:45.772775 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.773296 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.774422 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.774747 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.774827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.776217 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:45.776220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:45.776222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:45.776242 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:45.777563 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.778069 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.778073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.778074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.778258 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:45.778533 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:45.778536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:45.778538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:45.778699 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:45.778898 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.779081 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.779194 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260809 03:32:45.780130 7122 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46203
I20260809 03:32:45.780402 7122 heartbeater.cc:464] Registering TS with master...
I20260809 03:32:45.780900 7122 heartbeater.cc:511] Master 127.15.222.126:46203 requested a full tablet report, sending...
I20260809 03:32:45.781949 6412 ts_manager.cc:194] Registered new tserver with Master: d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345)
I20260809 03:32:45.782487 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:57683
I20260809 03:32:45.792583 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:32:45.796350 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.793136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.15.222.126:46203)
0809 03:32:45.793252 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.793256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.793446 (+ 190us) server_negotiation.cc:407] Connection header received
0809 03:32:45.793564 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.793566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.793653 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.793756 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:45.793940 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.794407 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.795202 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.795409 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.795472 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.795727 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:45.796078 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:45.796158 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.796190 (+ 32us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.796234 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:32:45.796918 6412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:32:45.800109 6412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:32:45.806939 6619 tablet_service.cc:1467] Processing CreateTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 (DEFAULT_TABLE table=test-table [id=c13543dba16e471f861de35f00342213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:45.806941 6819 tablet_service.cc:1467] Processing CreateTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 (DEFAULT_TABLE table=test-table [id=c13543dba16e471f861de35f00342213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:45.807559 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a56e8e2ad45ee871f2e9574b25191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:45.807559 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a56e8e2ad45ee871f2e9574b25191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:45.810294 6766 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Bootstrap starting.
I20260809 03:32:45.810294 6554 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap starting.
I20260809 03:32:45.810614 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.806197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34345 (local address 127.0.0.1:45532)
0809 03:32:45.806361 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:45.806375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:45.806441 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:45.807369 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:45.807372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:45.807391 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:45.807475 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.807481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.808521 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.808525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.809451 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:45.809461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.810266 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.810270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:45.810352 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.810372 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 03:32:45.810432 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 03:32:45.810501 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 03:32:45.810658 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:45.806614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.15.222.67:34345)
0809 03:32:45.806944 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:45.806952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:45.806971 (+ 19us) server_negotiation.cc:407] Connection header received
0809 03:32:45.807031 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:45.807035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:45.807220 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:45.807380 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:45.807570 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.808396 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.809569 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:45.810150 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:45.810257 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:45.810401 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:45.810455 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 03:32:45.810539 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":141,"threads_started":1}
I20260809 03:32:45.811115 6766 tablet_bootstrap.cc:654] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:45.811115 6554 tablet_bootstrap.cc:654] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:45.812505 6766 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: No bootstrap required, opened a new log
I20260809 03:32:45.812588 6766 ts_tablet_manager.cc:1397] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:45.812754 6554 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: No bootstrap required, opened a new log
I20260809 03:32:45.812831 6554 ts_tablet_manager.cc:1397] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:32:45.813186 6766 raft_consensus.cc:348] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.813203 6554 raft_consensus.cc:348] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.813297 6766 raft_consensus.cc:374] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:45.813301 6554 raft_consensus.cc:374] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:45.813337 6554 raft_consensus.cc:729] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaecdb9ef547fc9af17e9f4d331388, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.813336 6766 raft_consensus.cc:729] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8727f41110be460c9f1e0958fce0609c, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.813457 6766 consensus_queue.cc:260] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.813457 6554 consensus_queue.cc:260] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.813730 6554 ts_tablet_manager.cc:1428] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:32:45.813730 6766 ts_tablet_manager.cc:1428] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:32:45.814145 7016 tablet_service.cc:1467] Processing CreateTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 (DEFAULT_TABLE table=test-table [id=c13543dba16e471f861de35f00342213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:32:45.815292 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a56e8e2ad45ee871f2e9574b25191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:32:45.821702 7143 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Bootstrap starting.
I20260809 03:32:45.823654 7143 tablet_bootstrap.cc:654] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Neither blocks nor log segments found. Creating new log.
I20260809 03:32:45.824723 7143 log.cc:824] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:45.826359 7143 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: No bootstrap required, opened a new log
I20260809 03:32:45.826516 7143 ts_tablet_manager.cc:1397] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
W20260809 03:32:45.829106 6919 tablet.cc:2369] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:45.829222 7143 raft_consensus.cc:348] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.829487 7143 raft_consensus.cc:374] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:32:45.829558 7143 raft_consensus.cc:729] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a320d79a8841628143e55c65dda5f7, State: Initialized, Role: FOLLOWER
I20260809 03:32:45.829981 7143 consensus_queue.cc:260] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:45.830552 7122 heartbeater.cc:503] Master 127.15.222.126:46203 was elected leader, sending a full tablet report...
I20260809 03:32:45.830832 7143 ts_tablet_manager.cc:1428] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260809 03:32:45.891049 6722 tablet.cc:2369] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:32:46.014997 7123 tablet.cc:2369] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:32:46.041347 6928 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:46.041554 6928 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:46.042698 6928 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:46.046512 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.042882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34345 (local address 127.15.222.66:34803)
0809 03:32:46.042998 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.043013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:46.043110 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:46.043455 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:46.043459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:46.043481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:46.043601 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:46.043608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.044598 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.044601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:46.045369 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:46.045378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.046102 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.046105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:46.046211 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:46.046233 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:32:46.046300 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:32:46.046369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:32:46.046512 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.043044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34803 (local address 127.15.222.67:34345)
0809 03:32:46.043168 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:46.043172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:46.043187 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:32:46.043246 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:46.043249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:46.043298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:46.043434 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:46.043701 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.044490 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.045503 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.046003 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.046109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:46.046164 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:46.046325 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 03:32:46.046394 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:32:46.046785 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.043028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:34221 (local address 127.15.222.65:45759)
0809 03:32:46.043350 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:46.043354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:46.043446 (+ 92us) server_negotiation.cc:407] Connection header received
0809 03:32:46.043552 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:46.043555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:46.043644 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:46.043747 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:32:46.044071 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.044742 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.045696 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.046237 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.046346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:46.046503 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:46.046577 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 03:32:46.046660 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":75,"threads_started":1}
I20260809 03:32:46.047132 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.042882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.15.222.66:34221)
0809 03:32:46.043367 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.043380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:46.043472 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:46.043763 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:46.043765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:46.043782 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:32:46.043935 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:46.043943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.044842 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.044846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:46.045586 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:46.045594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.046363 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.046367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:46.046464 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:46.046485 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:32:46.046562 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 03:32:46.047023 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":125,"thread_start_us":110,"threads_started":1}
I20260809 03:32:46.047139 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a56e8e2ad45ee871f2e9574b25191" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a320d79a8841628143e55c65dda5f7" is_pre_election: true
I20260809 03:32:46.047470 7051 raft_consensus.cc:2454] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 0.
I20260809 03:32:46.047653 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a56e8e2ad45ee871f2e9574b25191" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaecdb9ef547fc9af17e9f4d331388" is_pre_election: true
I20260809 03:32:46.047928 6646 raft_consensus.cc:2454] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 0.
I20260809 03:32:46.047943 6757 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7; no voters:
I20260809 03:32:46.048205 6928 raft_consensus.cc:2781] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:32:46.048302 6928 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:32:46.048352 6928 raft_consensus.cc:3037] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:46.049415 6928 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:46.049754 6928 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 election: Requested vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759)
I20260809 03:32:46.050060 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a56e8e2ad45ee871f2e9574b25191" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a320d79a8841628143e55c65dda5f7"
I20260809 03:32:46.050060 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a56e8e2ad45ee871f2e9574b25191" candidate_uuid: "8727f41110be460c9f1e0958fce0609c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaecdb9ef547fc9af17e9f4d331388"
I20260809 03:32:46.050181 6646 raft_consensus.cc:3037] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:46.050181 7051 raft_consensus.cc:3037] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:32:46.051366 7051 raft_consensus.cc:2454] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 1.
I20260809 03:32:46.051366 6646 raft_consensus.cc:2454] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8727f41110be460c9f1e0958fce0609c in term 1.
I20260809 03:32:46.051638 6762 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388, 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7; no voters:
I20260809 03:32:46.051803 6928 raft_consensus.cc:2781] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:32:46.052019 6928 raft_consensus.cc:686] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 1 LEADER]: Becoming Leader. State: Replica: 8727f41110be460c9f1e0958fce0609c, State: Running, Role: LEADER
I20260809 03:32:46.052255 6928 consensus_queue.cc:237] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:46.055125 6411 catalog_manager.cc:5495] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8727f41110be460c9f1e0958fce0609c (127.15.222.66). New cstate: current_term: 1 leader_uuid: "8727f41110be460c9f1e0958fce0609c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } health_report { overall_health: UNKNOWN } } }
I20260809 03:32:46.101408 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.098493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.15.222.66:42071)
0809 03:32:46.098613 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:46.098617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:46.098629 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:32:46.098744 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:46.098747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:46.098800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:46.098887 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:32:46.099064 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.099541 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.100262 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:46.100458 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:46.100523 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:46.100701 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:32:46.101119 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:32:46.101199 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:46.101245 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:32:46.101295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:32:46.105082 6646 raft_consensus.cc:1261] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Refusing update from remote peer 8727f41110be460c9f1e0958fce0609c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:46.105086 7051 raft_consensus.cc:1261] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Refusing update from remote peer 8727f41110be460c9f1e0958fce0609c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:32:46.105820 7149 consensus_queue.cc:1035] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:46.106029 6928 consensus_queue.cc:1035] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:32:46.110736 7151 mvcc.cc:205] Tried to move back new op lower bound from 7316465115560542208 to 7316465115352477696. Current Snapshot: MvccSnapshot[applied={T|T < 7316465115560542208}]
I20260809 03:32:46.111997 7152 mvcc.cc:205] Tried to move back new op lower bound from 7316465115560542208 to 7316465115352477696. Current Snapshot: MvccSnapshot[applied={T|T < 7316465115560542208}]
I20260809 03:32:46.119751 6411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:46.119887 6411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:32:46.121975 6411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Sending DeleteTablet for 3 replicas of tablet c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:46.122725 7016 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:46 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:45532
I20260809 03:32:46.122733 6819 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:46 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260809 03:32:46.122925 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 6474
I20260809 03:32:46.122803 6619 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a56e8e2ad45ee871f2e9574b25191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:32:46 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:54012
I20260809 03:32:46.125977 7168 tablet_replica.cc:330] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: stopping tablet replica
I20260809 03:32:46.126281 7168 raft_consensus.cc:2229] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:32:46.126573 7168 raft_consensus.cc:2258] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:46.128918 7168 ts_tablet_manager.cc:1905] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:46.131688 7169 tablet_replica.cc:330] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: stopping tablet replica
I20260809 03:32:46.131847 7169 raft_consensus.cc:2229] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:32:46.131960 7168 ts_tablet_manager.cc:1918] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:46.132119 7168 log.cc:1196] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:46.132309 7169 raft_consensus.cc:2258] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:32:46.132480 7168 ts_tablet_manager.cc:1939] T c15a56e8e2ad45ee871f2e9574b25191 P d9a320d79a8841628143e55c65dda5f7: Deleting consensus metadata
I20260809 03:32:46.133126 7169 ts_tablet_manager.cc:1905] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:46.134114 6388 catalog_manager.cc:4841] TS d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): tablet c15a56e8e2ad45ee871f2e9574b25191 (table test-table [id=c13543dba16e471f861de35f00342213]) successfully deleted
W20260809 03:32:46.134482 6390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:45759 (error 108)
W20260809 03:32:46.135664 6729 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:46.135371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:45759 (local address 127.0.0.1:54026)
0809 03:32:46.135508 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.135578 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:45759: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 03:32:46.135872 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 6725
I20260809 03:32:46.136021 7169 ts_tablet_manager.cc:1918] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:46.136137 7169 log.cc:1196] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:32:46.135816 6390 catalog_manager.cc:4584] TS 56eaecdb9ef547fc9af17e9f4d331388 (127.15.222.65:45759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c15a56e8e2ad45ee871f2e9574b25191: Network error: Client connection negotiation failed: client connection to 127.15.222.65:45759: connect: Connection reset by peer (error 104)
I20260809 03:32:46.147140 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 6921
I20260809 03:32:46.147617 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.147331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42071 (local address 127.0.0.1:52240)
0809 03:32:46.147489 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.147537 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.66:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 03:32:46.161959 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 6364
I20260809 03:32:46.174049 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--webserver_interface=127.15.222.126
--webserver_port=42903
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:32:46.282706 7174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:46.282959 7174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:46.283005 7174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:46.286935 7174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:32:46.286990 7174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:46.287010 7174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:46.287039 7174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:32:46.287065 7174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:32:46.294318 7174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:46.295869 7174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:46.297029 7174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:46.305289 7181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.305281 7180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.305378 7183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:46.305442 7174 server_base.cc:1034] running on GCE node
I20260809 03:32:46.306253 7174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:46.307324 7174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:46.308485 7174 hybrid_clock.cc:648] HybridClock initialized: now 1786246366308456 us; error 45 us; skew 500 ppm
Aug 09 03:32:46 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:46.311184 7174 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:32:46.312017 7174 webserver.cc:460] Webserver started at http://127.15.222.126:42903/ using document root <none> and password file <none>
I20260809 03:32:46.312546 7174 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:46.312611 7174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:46.316397 7174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:32:46.318528 7196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.319690 7174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:32:46.319798 7174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a"
format_stamp: "Formatted at 2026-08-09 03:32:43 on dist-test-slave-cvwc"
server_key: "6f6602bf45e67b76a396c7ff335073b0"
server_key_iv: "c559e491833f06909f8b72bdae5aeaac"
server_key_version: "encryptionkey@0"
I20260809 03:32:46.320304 7174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:46.337204 7174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:46.339735 7174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:46.339888 7174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:46.346330 7174 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46203
I20260809 03:32:46.346577 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46203 every 8 connection(s)
I20260809 03:32:46.347313 7174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:32:46.349911 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 7174
I20260809 03:32:46.350364 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:45759
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=41415
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:46.352134 7274 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:32:46.353739 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap starting.
I20260809 03:32:46.370898 7274 log.cc:824] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:46.382504 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:46.382870 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a: Bootstrap complete.
I20260809 03:32:46.386881 7274 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:46.387252 7274 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Initialized, Role: FOLLOWER
I20260809 03:32:46.387715 7274 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:46.387846 7274 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:32:46.387917 7274 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:32:46.388011 7274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 2 FOLLOWER]: Advancing to term 3
I20260809 03:32:46.389834 7274 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:46.390148 7274 leader_election.cc:302] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b741e56b8b4ce3b8968ee5faa4af9a; no voters:
I20260809 03:32:46.390442 7274 leader_election.cc:288] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 03:32:46.390537 7281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 3 FOLLOWER]: Leader election won for term 3
I20260809 03:32:46.390707 7281 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [term 3 LEADER]: Becoming Leader. State: Replica: e4b741e56b8b4ce3b8968ee5faa4af9a, State: Running, Role: LEADER
I20260809 03:32:46.390995 7281 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } }
I20260809 03:32:46.391644 7274 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:32:46.391909 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b741e56b8b4ce3b8968ee5faa4af9a. Latest consensus state: current_term: 3 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:46.392016 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:46.392381 7286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:32:46.392419 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b741e56b8b4ce3b8968ee5faa4af9a" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46203 } } }
I20260809 03:32:46.392652 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b741e56b8b4ce3b8968ee5faa4af9a [sys.catalog]: This master's current role is: LEADER
I20260809 03:32:46.397459 7286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:32:46.398195 7286 catalog_manager.cc:1259] Loaded cluster ID: 4eda91d7033941cdbf47bdbfd464a207
I20260809 03:32:46.398265 7286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 03:32:46.399374 7286 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fecb517f9ad google::LogMessage::Fail() at ??:0
@ 0x7fecb518288a google::LogMessage::SendToLog() at ??:0
@ 0x7fecb517f4de google::LogMessage::Flush() at ??:0
@ 0x7fecb5180a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fecbfac2fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fecbfac3d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fecbfac0579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fecbfb04640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fecb73d13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fecb73d1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fecb73d33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fecb73c2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fecb8a066db start_thread at ??:0
@ 0x7fecb3c7d61f clone at ??:0
W20260809 03:32:46.472419 7276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:46.472651 7276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:46.472707 7276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:46.476657 7276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:46.476720 7276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:46.476795 7276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:32:46.482092 7276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:45759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=41415
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:46.483098 7276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:46.484112 7276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:46.493326 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.493331 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.493764 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:46.493999 7276 server_base.cc:1034] running on GCE node
I20260809 03:32:46.494365 7276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:46.494971 7276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:46.496148 7276 hybrid_clock.cc:648] HybridClock initialized: now 1786246366496128 us; error 27 us; skew 500 ppm
Aug 09 03:32:46 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:46.498778 7276 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:32:46.499648 7276 webserver.cc:460] Webserver started at http://127.15.222.65:41415/ using document root <none> and password file <none>
I20260809 03:32:46.500164 7276 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:46.500224 7276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:46.504034 7276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:32:46.506125 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.507400 7276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:46.507503 7276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "56eaecdb9ef547fc9af17e9f4d331388"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "769d2cd44e4d529c9610e4b959d6d16f"
server_key_iv: "47df86e659c23be56bf2d0f2fc395d0e"
server_key_version: "encryptionkey@0"
I20260809 03:32:46.508009 7276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:46.521607 7276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:46.524614 7276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:46.524772 7276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:46.525314 7276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:46.526400 7358 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:32:46.530310 7276 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:32:46.530362 7276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 03:32:46.530411 7276 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:32:46.531968 7276 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:32:46.532023 7276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 03:32:46.532297 7358 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap starting.
I20260809 03:32:46.542079 7276 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:45759
I20260809 03:32:46.542397 7513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:45759 every 8 connection(s)
I20260809 03:32:46.543259 7276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:32:46.547834 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 7276
I20260809 03:32:46.548319 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:42071
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=36729
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:46.552176 7358 log.cc:824] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Log is configured to *not* fsync() on all Append() calls
I20260809 03:32:46.561055 7358 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:32:46.561475 7358 tablet_bootstrap.cc:492] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Bootstrap complete.
I20260809 03:32:46.562829 7358 ts_tablet_manager.cc:1397] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.012s
I20260809 03:32:46.565891 7358 raft_consensus.cc:348] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:46.566421 7358 raft_consensus.cc:729] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaecdb9ef547fc9af17e9f4d331388, State: Initialized, Role: FOLLOWER
I20260809 03:32:46.566921 7358 consensus_queue.cc:260] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:46.567901 7358 ts_tablet_manager.cc:1428] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 03:32:46.601593 7519 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:32:46.545479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:46527)
0809 03:32:46.546045 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.546101 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:46.547098 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:46.600629 (+ 53531us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
W20260809 03:32:46.602190 7515 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46203 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46203: Client connection negotiation failed: client connection to 127.15.222.126:46203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:32:46.602954 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.602683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:38345)
0809 03:32:46.602814 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.602889 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 03:32:46.603564 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.603359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:33793)
0809 03:32:46.603463 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.603510 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 03:32:46.603720 7515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:32:46.667809 7520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:46.668051 7520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:46.668089 7520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:46.672120 7520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:46.672188 7520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:46.672266 7520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:32:46.677829 7520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:42071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=36729
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:46.678812 7520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:46.679876 7520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:46.689028 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.689114 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.689024 7538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:46.689193 7520 server_base.cc:1034] running on GCE node
I20260809 03:32:46.689698 7520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:46.690297 7520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:46.691448 7520 hybrid_clock.cc:648] HybridClock initialized: now 1786246366691434 us; error 30 us; skew 500 ppm
Aug 09 03:32:46 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:46.693964 7520 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:32:46.694759 7520 webserver.cc:460] Webserver started at http://127.15.222.66:36729/ using document root <none> and password file <none>
I20260809 03:32:46.695266 7520 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:46.695333 7520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:46.698968 7520 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:32:46.700910 7548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.702093 7520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:46.702194 7520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "8727f41110be460c9f1e0958fce0609c"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "a588cc46a81d404460be83ddd54b0e15"
server_key_iv: "a610b789f1e178f011702ef2baaf7196"
server_key_version: "encryptionkey@0"
I20260809 03:32:46.702709 7520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:46.712623 7520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:46.715101 7520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:46.715260 7520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:46.715783 7520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:46.716761 7558 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260809 03:32:46.720427 7558 ts_tablet_manager.cc:1852] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260809 03:32:46.720799 7558 ts_tablet_manager.cc:1905] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:32:46.724440 7558 ts_tablet_manager.cc:1918] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:32:46.724534 7558 log.cc:1196] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:46.724797 7558 ts_tablet_manager.cc:1939] T c15a56e8e2ad45ee871f2e9574b25191 P 8727f41110be460c9f1e0958fce0609c: Deleting consensus metadata
I20260809 03:32:46.725068 7520 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260809 03:32:46.725123 7520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260809 03:32:46.725189 7520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:46.725214 7520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.733407 7520 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42071
I20260809 03:32:46.733453 7706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42071 every 8 connection(s)
I20260809 03:32:46.734221 7520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:32:46.736958 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 7520
I20260809 03:32:46.737380 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:34345
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:46203
--webserver_port=38295
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:32:46.737550 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.736283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:45365)
0809 03:32:46.736728 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.736827 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
W20260809 03:32:46.737993 7707 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46203 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46203: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
I20260809 03:32:46.738561 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.738349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:44837)
0809 03:32:46.738453 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.738508 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 03:32:46.739133 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.738950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:35841)
0809 03:32:46.739032 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.739074 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 03:32:46.739352 7707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:32:46.845433 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:32:46.845763 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:32:46.845835 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:32:46.849984 7711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:32:46.850040 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:32:46.850116 7711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:32:46.855593 7711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:34345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=38295
--tserver_master_addrs=127.15.222.126:46203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:32:46.856561 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:32:46.857529 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:32:46.866504 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.866539 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:32:46.866528 7716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:32:46.866712 7711 server_base.cc:1034] running on GCE node
I20260809 03:32:46.867096 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:32:46.867679 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:32:46.868836 7711 hybrid_clock.cc:648] HybridClock initialized: now 1786246366868814 us; error 32 us; skew 500 ppm
Aug 09 03:32:46 dist-test-slave-cvwc krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:32:46.871327 7711 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:32:46.872138 7711 webserver.cc:460] Webserver started at http://127.15.222.67:38295/ using document root <none> and password file <none>
I20260809 03:32:46.872643 7711 fs_manager.cc:359] Metadata directory not provided
I20260809 03:32:46.872707 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:32:46.876288 7711 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:46.878225 7733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.879383 7711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:32:46.879484 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d9a320d79a8841628143e55c65dda5f7"
format_stamp: "Formatted at 2026-08-09 03:32:44 on dist-test-slave-cvwc"
server_key: "fcf383817ab0d1f18f093ab6a3f336f3"
server_key_iv: "376cab2cf49a811f3e550a13c46587f9"
server_key_version: "encryptionkey@0"
I20260809 03:32:46.879988 7711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:32:46.908612 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:32:46.911259 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:32:46.911424 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:32:46.912005 7711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:32:46.912796 7711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:32:46.912842 7711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.912902 7711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:32:46.912927 7711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:32:46.921221 7711 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34345
I20260809 03:32:46.921259 7900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34345 every 8 connection(s)
I20260809 03:32:46.922175 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:32:46.923985 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 7711
I20260809 03:32:46.926172 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.924174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:48465)
0809 03:32:46.924687 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.924832 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
W20260809 03:32:46.926606 7901 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46203 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46203: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
I20260809 03:32:46.927095 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.926901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:40621)
0809 03:32:46.926998 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.927043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 03:32:46.927610 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:46.927449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:41053)
0809 03:32:46.927527 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:46.927566 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:32:46.927757 7901 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:32:47.604799 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:47.604335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:49177)
0809 03:32:47.604648 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:47.604700 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260809 03:32:47.740195 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:47.739829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:38057)
0809 03:32:47.740057 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:47.740109 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260809 03:32:47.928570 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:47.928216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:42215)
0809 03:32:47.928422 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:47.928473 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260809 03:32:48.305044 7910 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:48.305269 7910 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:48.306401 7910 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
Aug 09 03:32:48 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.67@KRBTEST.COM
Aug 09 03:32:48 dist-test-slave-cvwc krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.66@KRBTEST.COM
I20260809 03:32:48.321130 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.306629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34345 (local address 127.15.222.65:52427)
0809 03:32:48.307189 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:48.307203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:48.307317 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:48.309576 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:48.309578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:48.309784 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:48.310051 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:48.310067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.311342 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.311345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:48.312870 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:48.312882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.313552 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.313554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:48.313688 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:48.314285 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:48.314300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:48.316066 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:48.318299 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:48.318305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:48.318321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:48.318813 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:48.319147 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:48.319150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:48.319152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:48.319339 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:48.320057 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:48.320064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:48.320401 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 03:32:48.320610 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 03:32:48.320993 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":318,"spinlock_wait_cycles":30848,"thread_start_us":81,"threads_started":1}
I20260809 03:32:48.321193 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.307170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:46843 (local address 127.15.222.66:42071)
0809 03:32:48.307497 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:48.307509 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:48.307547 (+ 38us) server_negotiation.cc:407] Connection header received
0809 03:32:48.308323 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:48.308330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:48.308770 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:48.309099 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:48.310225 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.311306 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.313747 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.314342 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.314476 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:48.316528 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:48.316534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:48.316549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:48.316583 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:48.318251 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:48.318892 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:48.318898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:48.318905 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:48.319054 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:48.319428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:48.319431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:48.319433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:48.319941 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:48.320072 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:48.320732 (+ 660us) server_negotiation.cc:299] Negotiation successful
0809 03:32:48.321011 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":68,"threads_started":1}
I20260809 03:32:48.321198 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.306781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:52427 (local address 127.15.222.67:34345)
0809 03:32:48.307131 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0809 03:32:48.307160 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 03:32:48.307286 (+ 126us) server_negotiation.cc:407] Connection header received
0809 03:32:48.308315 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:32:48.308324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:32:48.308770 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:32:48.309099 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:32:48.310209 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.311211 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.312980 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.313458 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.313581 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:48.316262 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:32:48.316266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:32:48.316278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:32:48.316306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:32:48.318186 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:48.318919 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:48.318924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:48.318928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:48.319054 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:32:48.319428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:32:48.319431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:32:48.319433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:32:48.319943 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:32:48.320072 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 03:32:48.320732 (+ 660us) server_negotiation.cc:299] Negotiation successful
0809 03:32:48.321011 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":133,"threads_started":1}
I20260809 03:32:48.321128 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.306990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42071 (local address 127.15.222.65:46843)
0809 03:32:48.307374 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:48.307389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:32:48.307503 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:32:48.309141 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:32:48.309146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:32:48.309525 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:32:48.310053 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:48.310067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.311419 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.311422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:48.312870 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:32:48.312890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:32:48.314453 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:32:48.314457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:32:48.314542 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:32:48.314931 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:32:48.314946 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:32:48.316340 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:32:48.318349 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:48.318352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:48.318354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:48.318790 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:48.319147 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:32:48.319150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:32:48.319152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:32:48.319339 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:32:48.320057 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:32:48.320064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:32:48.320401 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 03:32:48.320610 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 03:32:48.320928 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":193,"spinlock_wait_cycles":38784,"thread_start_us":84,"threads_started":1}
W20260809 03:32:48.323637 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:32:48.323836 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:48.323926 7351 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388; no voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7
I20260809 03:32:48.324172 7910 raft_consensus.cc:2726] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:32:48.605774 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.605480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:44565)
0809 03:32:48.605603 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:48.605654 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:32:48.741279 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.740868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:45135)
0809 03:32:48.741125 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:48.741180 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260809 03:32:48.929545 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:48.929164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:52997)
0809 03:32:48.929404 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:48.929460 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260809 03:32:49.606851 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:49.606458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:57605)
0809 03:32:49.606695 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:49.606753 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260809 03:32:49.742357 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:49.741970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:58943)
0809 03:32:49.742203 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:49.742254 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260809 03:32:49.891139 7922 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:49.891264 7922 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:49.891667 7922 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
W20260809 03:32:49.892477 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:32:49.892619 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:49.892668 7351 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388; no voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7
I20260809 03:32:49.892834 7922 raft_consensus.cc:2726] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:32:49.930559 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:49.930182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:38279)
0809 03:32:49.930419 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:49.930473 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260809 03:32:50.607908 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:50.607521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:57775)
0809 03:32:50.607759 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:50.607814 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260809 03:32:50.743330 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:50.742962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:46831)
0809 03:32:50.743193 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:50.743245 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260809 03:32:50.931562 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:50.931176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:50469)
0809 03:32:50.931415 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:50.931471 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260809 03:32:51.609156 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:51.608708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:48961)
0809 03:32:51.609002 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:51.609059 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260809 03:32:51.744374 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:51.743980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:39723)
0809 03:32:51.744228 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:51.744284 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260809 03:32:51.932591 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:51.932204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:41635)
0809 03:32:51.932441 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:51.932502 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260809 03:32:52.610308 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:52.609876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:51439)
0809 03:32:52.610156 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:52.610210 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260809 03:32:52.630090 7936 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:52.630218 7936 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:52.630568 7936 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
W20260809 03:32:52.631337 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:32:52.631482 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:52.631531 7351 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388; no voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7
I20260809 03:32:52.631700 7936 raft_consensus.cc:2726] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:32:52.745412 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:52.745014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:39469)
0809 03:32:52.745256 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:52.745306 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260809 03:32:52.933629 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:52.933242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:55475)
0809 03:32:52.933482 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:52.933537 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260809 03:32:53.611356 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:53.610971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:56573)
0809 03:32:53.611205 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:53.611259 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260809 03:32:53.746439 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:53.746020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:42011)
0809 03:32:53.746296 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:53.746349 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260809 03:32:53.934640 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:53.934241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:33287)
0809 03:32:53.934501 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:53.934553 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":69,"threads_started":1}
I20260809 03:32:54.612512 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:54.612112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:46809)
0809 03:32:54.612349 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:54.612413 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260809 03:32:54.747457 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:54.747074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:41077)
0809 03:32:54.747307 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:54.747361 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260809 03:32:54.784706 7949 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:32:54.784845 7949 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:32:54.785250 7949 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
W20260809 03:32:54.786084 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:32:54.786221 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:32:54.786281 7351 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388; no voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7
I20260809 03:32:54.786433 7949 raft_consensus.cc:2726] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:32:54.935701 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:54.935276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:44435)
0809 03:32:54.935542 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:54.935606 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260809 03:32:55.613485 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:55.613118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:42361)
0809 03:32:55.613338 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:55.613389 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260809 03:32:55.748481 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:55.748091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:45497)
0809 03:32:55.748328 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:55.748379 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260809 03:32:55.936834 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:55.936442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:57277)
0809 03:32:55.936685 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:55.936740 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260809 03:32:56.614594 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:56.614208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:56425)
0809 03:32:56.614442 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:56.614498 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260809 03:32:56.749505 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:56.749113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:54545)
0809 03:32:56.749359 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:56.749413 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260809 03:32:56.937963 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:56.937544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:41267)
0809 03:32:56.937807 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:56.937862 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
I20260809 03:32:57.615710 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:57.615289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:44451)
0809 03:32:57.615559 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:57.615613 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260809 03:32:57.750515 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:57.750115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:36267)
0809 03:32:57.750365 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:57.750425 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260809 03:32:57.938941 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:57.938581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:43111)
0809 03:32:57.938803 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:57.938854 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260809 03:32:58.616775 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:58.616390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:59147)
0809 03:32:58.616624 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:58.616678 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260809 03:32:58.751571 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:58.751166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:59473)
0809 03:32:58.751426 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:58.751478 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260809 03:32:58.939949 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:58.939576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:59639)
0809 03:32:58.939794 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:58.939847 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260809 03:32:59.617856 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:59.617452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:36773)
0809 03:32:59.617676 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:59.617757 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260809 03:32:59.752580 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:59.752200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:35569)
0809 03:32:59.752435 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:59.752489 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260809 03:32:59.941030 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:32:59.940620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:44191)
0809 03:32:59.940872 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 03:32:59.940937 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260809 03:33:00.618983 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:00.618559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:55993)
0809 03:33:00.618816 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:00.618875 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260809 03:33:00.665585 7975 raft_consensus.cc:482] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:00.665737 7975 raft_consensus.cc:504] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a320d79a8841628143e55c65dda5f7" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34345 } } peers { permanent_uuid: "8727f41110be460c9f1e0958fce0609c" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42071 } } peers { permanent_uuid: "56eaecdb9ef547fc9af17e9f4d331388" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 45759 } }
I20260809 03:33:00.666119 7975 leader_election.cc:288] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345), 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071)
W20260809 03:33:00.666898 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8727f41110be460c9f1e0958fce0609c (127.15.222.66:42071): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
W20260809 03:33:00.667016 7351 leader_election.cc:341] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9a320d79a8841628143e55c65dda5f7 (127.15.222.67:34345): Not found: Tablet not found: c15a56e8e2ad45ee871f2e9574b25191
I20260809 03:33:00.667052 7351 leader_election.cc:302] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56eaecdb9ef547fc9af17e9f4d331388; no voters: 8727f41110be460c9f1e0958fce0609c, d9a320d79a8841628143e55c65dda5f7
I20260809 03:33:00.667186 7975 raft_consensus.cc:2726] T c15a56e8e2ad45ee871f2e9574b25191 P 56eaecdb9ef547fc9af17e9f4d331388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:00.753605 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:00.753210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:50515)
0809 03:33:00.753457 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:00.753512 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260809 03:33:00.942179 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:00.941770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:48895)
0809 03:33:00.942022 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:00.942087 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260809 03:33:01.620194 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:01.619809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.65:42041)
0809 03:33:01.620047 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:01.620099 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260809 03:33:01.754761 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:01.754378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.66:36809)
0809 03:33:01.754614 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:01.754666 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260809 03:33:01.932761 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 7276
I20260809 03:33:01.943532 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:01.943122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46203 (local address 127.15.222.67:33673)
0809 03:33:01.943368 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:01.943439 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260809 03:33:01.946854 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 7520
I20260809 03:33:01.965062 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 7711
2026-08-09T03:33:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:33:02 dist-test-slave-cvwc krb5kdc[7987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:33:02 dist-test-slave-cvwc krb5kdc[7987](info): set up 2 sockets
Aug 09 03:33:02 dist-test-slave-cvwc krb5kdc[7987](info): commencing operation
krb5kdc: starting...
W20260809 03:33:03.116931 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:33:03.232472 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:03.339963 8015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:03.340195 8015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:03.340233 8015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:03.344110 8015 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:33:03.344151 8015 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:33:03.344164 8015 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:33:03.344177 8015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:33:03.344192 8015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:03.344214 8015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:03.344235 8015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:33:03.344254 8015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:33:03.349252 8015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:03.350162 8015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:03.351078 8015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:03.359308 8021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:03.359347 8015 server_base.cc:1034] running on GCE node
W20260809 03:33:03.359365 8022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:03.359346 8024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:03.360014 8015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:03.360986 8015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:03.362124 8015 hybrid_clock.cc:648] HybridClock initialized: now 1786246383362102 us; error 35 us; skew 500 ppm
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:03.368333 8015 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:33:03.369174 8015 webserver.cc:460] Webserver started at http://127.15.222.126:36987/ using document root <none> and password file <none>
I20260809 03:33:03.369684 8015 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:03.369779 8015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:03.369963 8015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:33:03.371886 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "7ac9e0ec4c584a9bb92065adae1e8274"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "0c720e3568b67107940358dd039871f2"
server_key_iv: "4f18a5c158c0f8938df495f0ccca12c2"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.372388 8015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "7ac9e0ec4c584a9bb92065adae1e8274"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "0c720e3568b67107940358dd039871f2"
server_key_iv: "4f18a5c158c0f8938df495f0ccca12c2"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.375885 8015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:33:03.378090 8034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.379242 8015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:33:03.379354 8015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "7ac9e0ec4c584a9bb92065adae1e8274"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "0c720e3568b67107940358dd039871f2"
server_key_iv: "4f18a5c158c0f8938df495f0ccca12c2"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.379451 8015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:03.397590 8015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:03.400174 8015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:03.400344 8015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:03.407066 8015 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46307
I20260809 03:33:03.407330 8109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46307 every 8 connection(s)
I20260809 03:33:03.407969 8015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:33:03.410636 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:03.415827 8110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap starting.
I20260809 03:33:03.417555 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8015
I20260809 03:33:03.417662 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:33:03.417948 16249 external_mini_cluster.cc:1428] Setting key 2658241f429c5b2dbe2972f729b25bd8
I20260809 03:33:03.418152 8110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:03.419018 8110 log.cc:824] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:03.420765 8110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: No bootstrap required, opened a new log
I20260809 03:33:03.424714 8110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:03.425102 8110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:03.425179 8110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Initialized, Role: FOLLOWER
I20260809 03:33:03.425678 8110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:03.425880 8110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:33:03.425953 8110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:33:03.426049 8110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:03.427062 8110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:03.427420 8110 leader_election.cc:302] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac9e0ec4c584a9bb92065adae1e8274; no voters:
I20260809 03:33:03.427795 8110 leader_election.cc:288] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:33:03.427956 8116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:33:03.428164 8116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Running, Role: LEADER
I20260809 03:33:03.428530 8116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:03.429196 8110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:33:03.429848 8116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac9e0ec4c584a9bb92065adae1e8274. Latest consensus state: current_term: 1 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:03.430014 8116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:03.430270 8117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:03.430423 8117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:03.430728 8123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:33:03.433022 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:03.418955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.15.222.126:46307)
0809 03:33:03.419360 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:03.419376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:03.419412 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:33:03.420001 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:03.420016 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:03.420304 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:03.420565 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:03.421478 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.422459 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.423605 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.424042 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.424149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:03.426961 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:03.426968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:03.426985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:03.427018 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:03.429043 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.429557 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.429563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.429569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.429706 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.430100 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.430103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.430105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.430646 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:03.431540 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:03.431698 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 03:33:03.432004 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":130,"threads_started":1}
I20260809 03:33:03.434108 8123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:33:03.441624 8123 catalog_manager.cc:1347] Generated new cluster ID: 91ee91af01b046c2a52883cc0962f75e
I20260809 03:33:03.441768 8123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:33:03.449786 8123 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:33:03.450947 8123 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:33:03.455207 8123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Generated new TSK 0
I20260809 03:33:03.455930 8123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:33:03.494001 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:03.605814 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:03.606060 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:03.606118 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:03.609987 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:03.610039 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:03.610116 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:33:03.616039 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:03.617285 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:03.618435 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:03.627818 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:03.627808 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:03.627808 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:03.628615 8148 server_base.cc:1034] running on GCE node
I20260809 03:33:03.628995 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:03.629591 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:03.630752 8148 hybrid_clock.cc:648] HybridClock initialized: now 1786246383630725 us; error 40 us; skew 500 ppm
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:03.633348 8148 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:33:03.634212 8148 webserver.cc:460] Webserver started at http://127.15.222.65:36101/ using document root <none> and password file <none>
I20260809 03:33:03.634716 8148 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:03.634774 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:03.634960 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:33:03.636957 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "568090aaa92b424d8132cb3f9f9ae561"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "edfe0e468fdc505438d7f23501467d98"
server_key_iv: "e8d5f9190f1a22eb798cf57d2c5e0b00"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.637486 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "568090aaa92b424d8132cb3f9f9ae561"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "edfe0e468fdc505438d7f23501467d98"
server_key_iv: "e8d5f9190f1a22eb798cf57d2c5e0b00"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.641185 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 03:33:03.643571 8170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.644805 8148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:03.644922 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "568090aaa92b424d8132cb3f9f9ae561"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "edfe0e468fdc505438d7f23501467d98"
server_key_iv: "e8d5f9190f1a22eb798cf57d2c5e0b00"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.645012 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:03.672747 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:03.676235 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:03.676426 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:03.677119 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:03.678140 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:03.678198 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.678258 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:03.678332 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 03:33:03.688793 8148 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:34435
I20260809 03:33:03.688803 8337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:34435 every 8 connection(s)
I20260809 03:33:03.689978 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:33:03.691744 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8148
I20260809 03:33:03.691870 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:33:03.692186 16249 external_mini_cluster.cc:1428] Setting key c7d4246ca5f67a7e12fdd81f2b6c57b2
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:33:03.704947 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:03.691722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:53827 (local address 127.15.222.126:46307)
0809 03:33:03.691849 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:03.691853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:03.692759 (+ 906us) server_negotiation.cc:407] Connection header received
0809 03:33:03.694205 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:03.694208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:03.694263 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:03.694336 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:03.695477 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.696155 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.697806 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.698132 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.698236 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:03.701275 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:03.701279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:03.701283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:03.701313 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:03.702994 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.703484 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.703488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.703489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.703568 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.703880 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.703884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.703885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.704071 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:03.704157 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:03.704636 (+ 479us) server_negotiation.cc:299] Negotiation successful
0809 03:33:03.704761 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
I20260809 03:33:03.705937 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:03.692137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:53827)
0809 03:33:03.692607 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:03.692659 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:03.694007 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:03.694519 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:03.694526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:03.694862 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:03.695326 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:03.695339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.696298 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.696302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:03.697655 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:03.697667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.698288 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.698292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:03.698508 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:03.699190 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:03.699210 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:03.701083 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:03.703106 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:03.703110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:03.703121 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:03.703381 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:03.703654 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:03.703657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:03.703659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:03.703791 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:03.704173 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:03.704179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:03.704414 (+ 235us) client_negotiation.cc:769] Sending connection context
0809 03:33:03.704577 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 03:33:03.704884 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260809 03:33:03.707105 8339 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:03.707346 8339 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:03.707872 8339 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:03.709422 8060 ts_manager.cc:194] Registered new tserver with Master: 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435)
I20260809 03:33:03.710625 8060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:53827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:33:03.721798 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:03.831794 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:03.832022 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:03.832069 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:03.835919 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:03.835966 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:03.836033 8347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:33:03.841017 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:03.841951 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:03.842904 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:03.851938 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:03.851963 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:03.851964 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:03.852236 8347 server_base.cc:1034] running on GCE node
I20260809 03:33:03.852626 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:03.853206 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:03.854354 8347 hybrid_clock.cc:648] HybridClock initialized: now 1786246383854335 us; error 29 us; skew 500 ppm
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:03.856897 8347 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:33:03.857682 8347 webserver.cc:460] Webserver started at http://127.15.222.66:36463/ using document root <none> and password file <none>
I20260809 03:33:03.858239 8347 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:03.858307 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:03.858500 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:33:03.860407 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "5ff61b057a584467b3c74853a40d8218"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "004b6249e65c8d56c693c8560301750a"
server_key_iv: "90bd3245920e206cef9a6f968ba6c64c"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.860924 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "5ff61b057a584467b3c74853a40d8218"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "004b6249e65c8d56c693c8560301750a"
server_key_iv: "90bd3245920e206cef9a6f968ba6c64c"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.864368 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:33:03.866550 8367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.867656 8347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:33:03.867758 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "5ff61b057a584467b3c74853a40d8218"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "004b6249e65c8d56c693c8560301750a"
server_key_iv: "90bd3245920e206cef9a6f968ba6c64c"
server_key_version: "encryptionkey@0"
I20260809 03:33:03.867844 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:03.890282 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:03.892916 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:03.893093 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:03.893652 8347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:03.894466 8347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:03.894513 8347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.894565 8347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:03.894595 8347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:03.902608 8347 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:33219
I20260809 03:33:03.902647 8534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:33219 every 8 connection(s)
I20260809 03:33:03.903457 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:33:03.907672 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8347
I20260809 03:33:03.907802 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:33:03.908136 16249 external_mini_cluster.cc:1428] Setting key 2a614863cc76a77cecb9e27c292b5f20
Aug 09 03:33:03 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:03.916700 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:03.905148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:44089 (local address 127.15.222.126:46307)
0809 03:33:03.905289 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:03.905293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:03.906078 (+ 785us) server_negotiation.cc:407] Connection header received
0809 03:33:03.907347 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:03.907350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:03.907403 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:03.907474 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:03.908894 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.909407 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.910538 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.910748 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.910835 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:03.913391 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:03.913394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:03.913396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:03.913420 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:03.914705 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.915283 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.915286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.915287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.915353 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:03.915690 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:03.915692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:03.915693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:03.915847 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:03.915922 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:03.916490 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 03:33:03.916581 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:33:03.917836 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:03.905503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:44089)
0809 03:33:03.905931 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:03.905971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:03.907151 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:03.907708 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:03.907717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:03.908183 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:03.908741 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:03.908756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.909524 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.909527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:03.910424 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:03.910433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:03.910879 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:03.910883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:03.911035 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:03.911607 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:03.911627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:03.913235 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:03.914827 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:03.914834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:03.914849 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:03.915194 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:03.915429 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:03.915432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:03.915435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:03.915609 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:03.915942 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:03.915949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:03.916249 (+ 300us) client_negotiation.cc:769] Sending connection context
0809 03:33:03.916472 (+ 223us) client_negotiation.cc:240] Negotiation successful
0809 03:33:03.916798 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:33:03.918919 8535 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:03.919130 8535 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:03.919571 8535 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:03.920531 8060 ts_manager.cc:194] Registered new tserver with Master: 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219)
I20260809 03:33:03.921066 8060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:44089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:33:03.934331 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:04.043350 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:04.043607 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:04.043666 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:04.047600 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:04.047652 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:04.047724 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:33:04.053000 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:04.053975 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:04.054943 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:04.064011 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:04.064069 8540 server_base.cc:1034] running on GCE node
W20260809 03:33:04.064023 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:04.064070 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:04.064558 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:04.065107 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:04.066246 8540 hybrid_clock.cc:648] HybridClock initialized: now 1786246384066213 us; error 44 us; skew 500 ppm
Aug 09 03:33:04 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:04.068864 8540 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:33:04.069743 8540 webserver.cc:460] Webserver started at http://127.15.222.67:45035/ using document root <none> and password file <none>
I20260809 03:33:04.070278 8540 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:04.070345 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:04.070555 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:33:04.072669 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "d6ca3f7207274b17b12cf3c516355a91"
format_stamp: "Formatted at 2026-08-09 03:33:04 on dist-test-slave-cvwc"
server_key: "1795a4aa72f5aaa322faff02acc9bf34"
server_key_iv: "5b295ec45cf8a114bdd5ef6ec748a157"
server_key_version: "encryptionkey@0"
I20260809 03:33:04.073201 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "d6ca3f7207274b17b12cf3c516355a91"
format_stamp: "Formatted at 2026-08-09 03:33:04 on dist-test-slave-cvwc"
server_key: "1795a4aa72f5aaa322faff02acc9bf34"
server_key_iv: "5b295ec45cf8a114bdd5ef6ec748a157"
server_key_version: "encryptionkey@0"
I20260809 03:33:04.076715 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:33:04.079088 8562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:04.080233 8540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:33:04.080339 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d6ca3f7207274b17b12cf3c516355a91"
format_stamp: "Formatted at 2026-08-09 03:33:04 on dist-test-slave-cvwc"
server_key: "1795a4aa72f5aaa322faff02acc9bf34"
server_key_iv: "5b295ec45cf8a114bdd5ef6ec748a157"
server_key_version: "encryptionkey@0"
I20260809 03:33:04.080440 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:04.094627 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:04.097476 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:04.097635 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:04.098206 8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:04.098992 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:04.099038 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:04.099088 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:04.099116 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:04.107295 8540 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34273
I20260809 03:33:04.107321 8736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34273 every 8 connection(s)
I20260809 03:33:04.108119 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:33:04.109525 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8540
I20260809 03:33:04.109643 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:33:04.109987 16249 external_mini_cluster.cc:1428] Setting key 3dbf8e8058df808908d0d52886e3951e
Aug 09 03:33:04 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:04.120934 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.109819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:59825 (local address 127.15.222.126:46307)
0809 03:33:04.109985 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.109990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.110829 (+ 839us) server_negotiation.cc:407] Connection header received
0809 03:33:04.112386 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.112389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.112442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.112514 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:04.113645 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.114225 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.115313 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.115535 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.115620 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.117871 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:04.117874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:04.117877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:04.117910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:04.119056 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:04.119512 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:04.119516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:04.119518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:04.119595 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:04.119906 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:04.119909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:04.119911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:04.120086 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:04.120171 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.120637 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.120765 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:33:04.121639 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.110220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:59825)
0809 03:33:04.110687 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.110742 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.112183 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.112687 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.112694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.113033 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:04.113501 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.113515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.114340 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.114344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.115194 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.115204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.115666 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.115670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.115830 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.116351 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:04.116367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:04.117693 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:04.119159 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:04.119165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:04.119179 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:04.119422 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:04.119676 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:04.119679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:04.119681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:04.119812 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:04.120182 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:04.120188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:04.120418 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.120578 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.120883 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260809 03:33:04.122635 8737 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:04.122870 8737 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:04.123425 8737 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:04.124361 8060 ts_manager.cc:194] Registered new tserver with Master: d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
I20260809 03:33:04.124902 8060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:59825
I20260809 03:33:04.135052 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:33:04.143442 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.136827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.15.222.126:46307)
0809 03:33:04.136931 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.136935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.137098 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:33:04.137227 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.137230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.137272 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.137328 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:04.138051 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.138552 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.139372 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.139592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.139675 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.140832 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:04.140835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:04.140838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:04.140862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:04.141923 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:04.142286 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:04.142289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:04.142291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:04.142364 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:04.142609 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:04.142613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:04.142614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:04.142821 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:04.142941 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.143188 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.143292 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:33:04.145792 8060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:33:04.148731 8060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:33:04.162638 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.157195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.15.222.66:33219)
0809 03:33:04.158179 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.158191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.158211 (+ 20us) server_negotiation.cc:407] Connection header received
0809 03:33:04.158281 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.158285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.158462 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.158616 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:04.158850 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.159989 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.161316 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.161969 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.162213 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.162394 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.162493 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":134,"threads_started":1}
I20260809 03:33:04.162632 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.157052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:33219 (local address 127.0.0.1:43422)
0809 03:33:04.157869 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.157881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.157964 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.158595 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.158598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.158616 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:04.158716 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.158722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.160133 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.160136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.161207 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.161216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.162089 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.162206 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.162242 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.162348 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.162493 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":59,"thread_start_us":92,"threads_started":1}
I20260809 03:33:04.162933 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.157120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.15.222.65:34435)
0809 03:33:04.157698 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.157901 (+ 203us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.157928 (+ 27us) server_negotiation.cc:407] Connection header received
0809 03:33:04.158001 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.158008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.158212 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.158388 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:04.158925 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.159770 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.161346 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.162056 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.162288 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.162703 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.162791 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":107,"threads_started":1}
I20260809 03:33:04.163090 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.156985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:34435 (local address 127.0.0.1:60048)
0809 03:33:04.157457 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.157504 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.157629 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.158430 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.158465 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.158506 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:04.158736 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.158745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.159935 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.159938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.161214 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.161226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.162291 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.162378 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.162427 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.162498 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.162567 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":28,"thread_start_us":104,"threads_started":1}
I20260809 03:33:04.164136 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.156404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34273 (local address 127.0.0.1:44338)
0809 03:33:04.157458 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.157504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.157629 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.159130 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.159133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.159154 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:04.159230 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.159235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.160511 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.160514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.161643 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.161652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.162940 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.163839 (+ 896us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.163863 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.163944 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.164011 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":49,"spinlock_wait_cycles":21248,"thread_start_us":58,"threads_started":1}
I20260809 03:33:04.164572 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.156542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.15.222.67:34273)
0809 03:33:04.156862 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.156874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.157785 (+ 911us) server_negotiation.cc:407] Connection header received
0809 03:33:04.157862 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.158805 (+ 943us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.158993 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.159140 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:04.159342 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.160365 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.161803 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.162546 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.162632 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.163210 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.164348 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.164438 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":122,"threads_started":1}
I20260809 03:33:04.165437 8439 tablet_service.cc:1467] Processing CreateTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 (DEFAULT_TABLE table=test-table [id=72311f800d984f89a43592072c8754fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:04.165534 8236 tablet_service.cc:1467] Processing CreateTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 (DEFAULT_TABLE table=test-table [id=72311f800d984f89a43592072c8754fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:04.166390 8439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181c4c3953914c378ec6ea15a8ea7606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:04.166388 8236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181c4c3953914c378ec6ea15a8ea7606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:04.172912 8767 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap starting.
I20260809 03:33:04.172839 8637 tablet_service.cc:1467] Processing CreateTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 (DEFAULT_TABLE table=test-table [id=72311f800d984f89a43592072c8754fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:04.173741 8637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181c4c3953914c378ec6ea15a8ea7606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:04.174759 8767 tablet_bootstrap.cc:654] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:04.175807 8767 log.cc:824] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:04.176229 8764 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Bootstrap starting.
I20260809 03:33:04.177903 8767 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: No bootstrap required, opened a new log
I20260809 03:33:04.178153 8767 ts_tablet_manager.cc:1397] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:33:04.178776 8764 tablet_bootstrap.cc:654] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:04.179306 8770 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Bootstrap starting.
I20260809 03:33:04.179579 8764 log.cc:824] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:04.181098 8770 tablet_bootstrap.cc:654] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:04.181546 8764 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: No bootstrap required, opened a new log
I20260809 03:33:04.181744 8764 ts_tablet_manager.cc:1397] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:33:04.181761 8767 raft_consensus.cc:348] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.182094 8767 raft_consensus.cc:374] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:04.182174 8767 raft_consensus.cc:729] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568090aaa92b424d8132cb3f9f9ae561, State: Initialized, Role: FOLLOWER
I20260809 03:33:04.182202 8770 log.cc:824] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:04.182622 8767 consensus_queue.cc:260] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.183573 8767 ts_tablet_manager.cc:1428] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:33:04.183955 8770 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: No bootstrap required, opened a new log
I20260809 03:33:04.184015 8339 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
I20260809 03:33:04.184168 8770 ts_tablet_manager.cc:1397] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:33:04.185153 8764 raft_consensus.cc:348] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.185429 8764 raft_consensus.cc:374] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:04.185496 8764 raft_consensus.cc:729] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff61b057a584467b3c74853a40d8218, State: Initialized, Role: FOLLOWER
I20260809 03:33:04.185957 8764 consensus_queue.cc:260] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.186611 8535 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
I20260809 03:33:04.187767 8764 ts_tablet_manager.cc:1428] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260809 03:33:04.188076 8770 raft_consensus.cc:348] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.188369 8770 raft_consensus.cc:374] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:04.188462 8770 raft_consensus.cc:729] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ca3f7207274b17b12cf3c516355a91, State: Initialized, Role: FOLLOWER
I20260809 03:33:04.188860 8770 consensus_queue.cc:260] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.189785 8737 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
I20260809 03:33:04.189872 8770 ts_tablet_manager.cc:1428] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260809 03:33:04.191109 8340 tablet.cc:2369] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:33:04.225828 8774 raft_consensus.cc:482] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:04.225987 8774 raft_consensus.cc:504] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.226982 8774 leader_election.cc:288] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273), 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219)
I20260809 03:33:04.231794 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.227669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:52675 (local address 127.15.222.66:33219)
0809 03:33:04.227790 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.227795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.227901 (+ 106us) server_negotiation.cc:407] Connection header received
0809 03:33:04.227977 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.227980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.228020 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.228081 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:04.228284 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.229008 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.230576 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.231246 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.231342 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.231382 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.231613 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.231679 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:33:04.231810 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.227565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:33219 (local address 127.15.222.65:52675)
0809 03:33:04.227809 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.227821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.227902 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.228101 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.228104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.228121 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:04.228197 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.228201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.229080 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.229083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.230462 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.230474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.231347 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.231351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.231480 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.231518 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.231593 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.231667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":15,"thread_start_us":94,"threads_started":1}
I20260809 03:33:04.232046 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.227287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:60907 (local address 127.15.222.67:34273)
0809 03:33:04.227411 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.227415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.227522 (+ 107us) server_negotiation.cc:407] Connection header received
0809 03:33:04.227710 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.227713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.227762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.227844 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:04.228116 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.228879 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.230576 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.231198 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.231312 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.231365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.231860 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.231940 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:33:04.232239 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.227200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34273 (local address 127.15.222.65:60907)
0809 03:33:04.227446 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.227462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:04.227562 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:04.227867 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:04.227871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:04.227901 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:04.228020 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.228026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.228972 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.228992 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.230464 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:04.230474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.231447 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.231451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:04.231607 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.231633 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 03:33:04.231703 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:33:04.231776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":158,"thread_start_us":88,"threads_started":1}
I20260809 03:33:04.232347 8468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181c4c3953914c378ec6ea15a8ea7606" candidate_uuid: "568090aaa92b424d8132cb3f9f9ae561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff61b057a584467b3c74853a40d8218" is_pre_election: true
I20260809 03:33:04.232643 8468 raft_consensus.cc:2454] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568090aaa92b424d8132cb3f9f9ae561 in term 0.
I20260809 03:33:04.232796 8667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181c4c3953914c378ec6ea15a8ea7606" candidate_uuid: "568090aaa92b424d8132cb3f9f9ae561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ca3f7207274b17b12cf3c516355a91" is_pre_election: true
I20260809 03:33:04.233075 8667 raft_consensus.cc:2454] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568090aaa92b424d8132cb3f9f9ae561 in term 0.
I20260809 03:33:04.233135 8176 leader_election.cc:302] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561, 5ff61b057a584467b3c74853a40d8218; no voters:
I20260809 03:33:04.233402 8774 raft_consensus.cc:2781] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:33:04.233482 8774 raft_consensus.cc:482] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:33:04.233532 8774 raft_consensus.cc:3037] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:04.234522 8774 raft_consensus.cc:504] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.234817 8774 leader_election.cc:288] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 1 election: Requested vote from peers d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273), 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219)
I20260809 03:33:04.235114 8667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181c4c3953914c378ec6ea15a8ea7606" candidate_uuid: "568090aaa92b424d8132cb3f9f9ae561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ca3f7207274b17b12cf3c516355a91"
I20260809 03:33:04.235198 8468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181c4c3953914c378ec6ea15a8ea7606" candidate_uuid: "568090aaa92b424d8132cb3f9f9ae561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff61b057a584467b3c74853a40d8218"
I20260809 03:33:04.235256 8667 raft_consensus.cc:3037] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:04.235307 8468 raft_consensus.cc:3037] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:04.236478 8667 raft_consensus.cc:2454] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568090aaa92b424d8132cb3f9f9ae561 in term 1.
I20260809 03:33:04.236478 8468 raft_consensus.cc:2454] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568090aaa92b424d8132cb3f9f9ae561 in term 1.
I20260809 03:33:04.236797 8176 leader_election.cc:302] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561, d6ca3f7207274b17b12cf3c516355a91; no voters:
I20260809 03:33:04.236982 8774 raft_consensus.cc:2781] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:33:04.237177 8774 raft_consensus.cc:686] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 LEADER]: Becoming Leader. State: Replica: 568090aaa92b424d8132cb3f9f9ae561, State: Running, Role: LEADER
I20260809 03:33:04.237490 8774 consensus_queue.cc:237] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:04.240336 8059 catalog_manager.cc:5495] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "568090aaa92b424d8132cb3f9f9ae561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } health_report { overall_health: UNKNOWN } } }
I20260809 03:33:04.257346 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.254478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.15.222.65:34435)
0809 03:33:04.254595 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:04.254598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:04.254610 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:33:04.254716 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:04.254718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:04.254763 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:04.254839 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:33:04.255024 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.255492 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.256226 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:04.256429 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:04.256491 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:04.256696 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:33:04.257062 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:33:04.257141 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:04.257190 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 03:33:04.257232 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:33:04.262614 8468 raft_consensus.cc:1261] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Refusing update from remote peer 568090aaa92b424d8132cb3f9f9ae561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:33:04.262614 8667 raft_consensus.cc:1261] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Refusing update from remote peer 568090aaa92b424d8132cb3f9f9ae561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:33:04.263273 8781 consensus_queue.cc:1035] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:33:04.263441 8774 consensus_queue.cc:1035] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:33:04.278110 8059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:33:04.278290 8059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:33:04.280676 8059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Sending DeleteTablet for 3 replicas of tablet 181c4c3953914c378ec6ea15a8ea7606
I20260809 03:33:04.281476 8236 tablet_service.cc:1514] Processing DeleteTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:04 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:60048
I20260809 03:33:04.281576 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8148
I20260809 03:33:04.281523 8637 tablet_service.cc:1514] Processing DeleteTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:04 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:44338
I20260809 03:33:04.281632 8439 tablet_service.cc:1514] Processing DeleteTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:04 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:43422
I20260809 03:33:04.289422 8802 tablet_replica.cc:330] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: stopping tablet replica
I20260809 03:33:04.289746 8802 raft_consensus.cc:2229] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:33:04.290122 8802 raft_consensus.cc:2258] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:04.291991 8802 ts_tablet_manager.cc:1905] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:33:04.292282 8803 tablet_replica.cc:330] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: stopping tablet replica
I20260809 03:33:04.292547 8803 raft_consensus.cc:2229] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:33:04.292824 8803 raft_consensus.cc:2258] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:04.293989 8803 ts_tablet_manager.cc:1905] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:33:04.294731 8802 ts_tablet_manager.cc:1918] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:33:04.294832 8802 log.cc:1196] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/181c4c3953914c378ec6ea15a8ea7606
I20260809 03:33:04.295176 8802 ts_tablet_manager.cc:1939] T 181c4c3953914c378ec6ea15a8ea7606 P d6ca3f7207274b17b12cf3c516355a91: Deleting consensus metadata
I20260809 03:33:04.296078 8803 ts_tablet_manager.cc:1918] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:33:04.296161 8803 log.cc:1196] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/181c4c3953914c378ec6ea15a8ea7606
I20260809 03:33:04.296371 8039 catalog_manager.cc:4841] TS d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): tablet 181c4c3953914c378ec6ea15a8ea7606 (table test-table [id=72311f800d984f89a43592072c8754fc]) successfully deleted
I20260809 03:33:04.296437 8803 ts_tablet_manager.cc:1939] T 181c4c3953914c378ec6ea15a8ea7606 P 5ff61b057a584467b3c74853a40d8218: Deleting consensus metadata
I20260809 03:33:04.297372 8039 catalog_manager.cc:4841] TS 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): tablet 181c4c3953914c378ec6ea15a8ea7606 (table test-table [id=72311f800d984f89a43592072c8754fc]) successfully deleted
W20260809 03:33:04.298161 8042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:34435 (error 108)
I20260809 03:33:04.298192 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8347
I20260809 03:33:04.303433 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:04.303032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:34435 (local address 127.0.0.1:60068)
0809 03:33:04.303247 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:04.303333 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260809 03:33:04.303640 8042 catalog_manager.cc:4584] TS 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 181c4c3953914c378ec6ea15a8ea7606: Network error: Client connection negotiation failed: client connection to 127.15.222.65:34435: connect: Connection refused (error 111)
I20260809 03:33:04.309585 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8540
I20260809 03:33:04.317687 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8015
I20260809 03:33:24.326637 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--webserver_interface=127.15.222.126
--webserver_port=36987
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:24.436105 8806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:24.436339 8806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:24.436383 8806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:24.440325 8806 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:33:24.440384 8806 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:33:24.440400 8806 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:33:24.440412 8806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:33:24.440429 8806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:24.440447 8806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:24.440477 8806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:33:24.440507 8806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:33:24.445946 8806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:24.446910 8806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:24.447870 8806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:24.460430 8826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:24.460417 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:24.460472 8822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:24.460809 8806 server_base.cc:1034] running on GCE node
I20260809 03:33:24.461409 8806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:24.462540 8806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:24.463694 8806 hybrid_clock.cc:648] HybridClock initialized: now 1786246404463675 us; error 32 us; skew 500 ppm
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:24.466408 8806 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:33:24.467399 8806 webserver.cc:460] Webserver started at http://127.15.222.126:36987/ using document root <none> and password file <none>
I20260809 03:33:24.467927 8806 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:24.467999 8806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:24.471709 8806 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:33:24.473678 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:24.474895 8806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:24.475018 8806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "7ac9e0ec4c584a9bb92065adae1e8274"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "0c720e3568b67107940358dd039871f2"
server_key_iv: "4f18a5c158c0f8938df495f0ccca12c2"
server_key_version: "encryptionkey@0"
I20260809 03:33:24.475602 8806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:24.494535 8806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:24.497295 8806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:24.497478 8806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:24.504241 8806 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46307
I20260809 03:33:24.504518 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46307 every 8 connection(s)
I20260809 03:33:24.505324 8806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:33:24.509569 8914 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:33:24.511209 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap starting.
I20260809 03:33:24.513417 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8806
I20260809 03:33:24.513834 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:34435
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=36101
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:33:24.527752 8914 log.cc:824] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:24.536525 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:33:24.536859 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap complete.
I20260809 03:33:24.540753 8914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:24.541155 8914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Initialized, Role: FOLLOWER
I20260809 03:33:24.541636 8914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:24.541795 8914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:33:24.541872 8914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:33:24.541975 8914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 1 FOLLOWER]: Advancing to term 2
I20260809 03:33:24.543792 8914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:24.544111 8914 leader_election.cc:302] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac9e0ec4c584a9bb92065adae1e8274; no voters:
I20260809 03:33:24.544400 8914 leader_election.cc:288] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 03:33:24.544529 8920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 03:33:24.544771 8920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 LEADER]: Becoming Leader. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Running, Role: LEADER
I20260809 03:33:24.545130 8920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:24.545553 8914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:33:24.546548 8921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:24.546669 8921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:24.547324 8936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:33:24.547427 8923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac9e0ec4c584a9bb92065adae1e8274. Latest consensus state: current_term: 2 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:24.547513 8923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:24.552030 8936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:33:24.552680 8936 catalog_manager.cc:1259] Loaded cluster ID: 91ee91af01b046c2a52883cc0962f75e
I20260809 03:33:24.552742 8936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:33:24.555231 8936 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:33:24.555939 8936 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Loaded TSK: 0
I20260809 03:33:24.570717 8936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Generated new TSK 1
I20260809 03:33:24.570982 8936 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:33:24.626791 8917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:24.627084 8917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:24.627143 8917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:24.631096 8917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:24.631155 8917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:24.631232 8917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:33:24.636381 8917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=36101
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:24.637372 8917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:24.638486 8917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:24.648995 8961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:24.649006 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:24.649255 8917 server_base.cc:1034] running on GCE node
W20260809 03:33:24.649518 8963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:24.649973 8917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:24.650585 8917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:24.651744 8917 hybrid_clock.cc:648] HybridClock initialized: now 1786246404651717 us; error 45 us; skew 500 ppm
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:24.654656 8917 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:33:24.655764 8917 webserver.cc:460] Webserver started at http://127.15.222.65:36101/ using document root <none> and password file <none>
I20260809 03:33:24.656504 8917 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:24.656618 8917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:24.661935 8917 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:33:24.664681 8972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:24.666288 8917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:33:24.666419 8917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "568090aaa92b424d8132cb3f9f9ae561"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "edfe0e468fdc505438d7f23501467d98"
server_key_iv: "e8d5f9190f1a22eb798cf57d2c5e0b00"
server_key_version: "encryptionkey@0"
I20260809 03:33:24.667145 8917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:24.696699 8917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:24.700299 8917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:24.700507 8917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:24.701226 8917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:24.702512 8985 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:33:24.706558 8917 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:33:24.706621 8917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260809 03:33:24.706676 8917 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:33:24.708349 8917 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:33:24.708393 8917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 03:33:24.708755 8985 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap starting.
I20260809 03:33:24.717633 8917 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:34435
I20260809 03:33:24.717775 9140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:34435 every 8 connection(s)
I20260809 03:33:24.718600 8917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:33:24.721176 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 8917
I20260809 03:33:24.721647 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:33219
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=36463
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:33:24.727952 8985 log.cc:824] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:24.736873 8985 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:33:24.736967 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.720676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:47841)
0809 03:33:24.721265 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:24.721335 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:24.722387 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:24.723903 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:24.723912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:24.724369 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:24.724979 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.724996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.727051 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.727055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.728436 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.728449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.729348 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.729353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.729551 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.730158 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:24.730176 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:24.731643 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:24.733752 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.733759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.733772 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.734045 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.734415 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.734418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.734420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.734568 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.735220 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:24.735228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:24.735483 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 03:33:24.735659 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 03:33:24.736008 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
I20260809 03:33:24.737006 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.720617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:47841 (local address 127.15.222.126:46307)
0809 03:33:24.720947 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:24.720966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:24.721457 (+ 491us) server_negotiation.cc:407] Connection header received
0809 03:33:24.722939 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:24.722960 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:24.723422 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:24.723812 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:24.725178 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.726890 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.728580 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.729177 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.729363 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.731843 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:24.731848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:24.731861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:24.731891 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:24.733592 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.734167 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.734172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.734176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.734315 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.734676 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.734678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.734679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.735030 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:24.735252 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:24.735814 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 03:33:24.736096 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
I20260809 03:33:24.737320 8985 tablet_bootstrap.cc:492] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap complete.
I20260809 03:33:24.738867 8985 ts_tablet_manager.cc:1397] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.011s
I20260809 03:33:24.739696 9142 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:24.739926 9142 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:24.740411 9142 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:24.741737 8985 raft_consensus.cc:348] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:24.742255 8985 raft_consensus.cc:729] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568090aaa92b424d8132cb3f9f9ae561, State: Initialized, Role: FOLLOWER
I20260809 03:33:24.742789 8864 ts_manager.cc:194] Registered new tserver with Master: 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435)
I20260809 03:33:24.742808 8985 consensus_queue.cc:260] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } } peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } }
I20260809 03:33:24.743803 8985 ts_tablet_manager.cc:1428] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260809 03:33:24.746327 8864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:47841
I20260809 03:33:24.749042 9142 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.65@KRBTEST.COM
I20260809 03:33:24.756657 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.745730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.15.222.65:34435)
0809 03:33:24.746471 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:24.746489 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:24.746518 (+ 29us) server_negotiation.cc:407] Connection header received
0809 03:33:24.746590 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:24.746595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:24.746790 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:24.746944 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:24.747609 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.748655 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.750439 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.750894 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.750994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.753369 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:24.753378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:24.753393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:24.753415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:24.754879 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.755381 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.755395 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.755397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.755468 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.755717 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.755719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.755720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.755936 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:24.756038 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:24.756361 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 03:33:24.756537 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":110,"threads_started":1}
I20260809 03:33:24.756664 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.745630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:34435 (local address 127.0.0.1:33140)
0809 03:33:24.746214 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:24.746301 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:24.746453 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:24.746948 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:24.746954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:24.747361 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:24.747495 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.747504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.749384 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.749387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.750323 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.750336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.751020 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.751024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.751131 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.751684 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:24.751699 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:24.753237 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:24.754985 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.754994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.755000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.755289 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.755543 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.755546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.755548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.755641 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.756037 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:24.756042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:24.756165 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 03:33:24.756308 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 03:33:24.756499 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":128,"threads_started":1}
I20260809 03:33:24.757696 9048 tablet_service.cc:1514] Processing DeleteTablet for tablet 181c4c3953914c378ec6ea15a8ea7606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:04 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260809 03:33:24.758585 9155 tablet_replica.cc:330] stopping tablet replica
I20260809 03:33:24.758872 9155 raft_consensus.cc:2229] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:33:24.759114 9155 raft_consensus.cc:2258] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:24.760102 9155 ts_tablet_manager.cc:1905] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:33:24.763746 9155 ts_tablet_manager.cc:1918] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:33:24.763840 9155 log.cc:1196] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal/wals/181c4c3953914c378ec6ea15a8ea7606
I20260809 03:33:24.764133 9155 ts_tablet_manager.cc:1939] T 181c4c3953914c378ec6ea15a8ea7606 P 568090aaa92b424d8132cb3f9f9ae561: Deleting consensus metadata
I20260809 03:33:24.765925 8842 catalog_manager.cc:4841] TS 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435): tablet 181c4c3953914c378ec6ea15a8ea7606 (table test-table [id=72311f800d984f89a43592072c8754fc]) successfully deleted
W20260809 03:33:24.842790 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:24.843036 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:24.843077 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:24.847865 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:24.847936 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:24.848018 9147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:33:24.854885 9147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:33219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=36463
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:24.855849 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:24.856863 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:24.866017 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:24.866089 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:24.866110 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:24.866417 9147 server_base.cc:1034] running on GCE node
I20260809 03:33:24.866746 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:24.867353 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:24.868513 9147 hybrid_clock.cc:648] HybridClock initialized: now 1786246404868486 us; error 37 us; skew 500 ppm
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:24.871090 9147 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:33:24.871925 9147 webserver.cc:460] Webserver started at http://127.15.222.66:36463/ using document root <none> and password file <none>
I20260809 03:33:24.872457 9147 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:24.872535 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:24.876283 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:24.878234 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:24.879550 9147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:24.879654 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "5ff61b057a584467b3c74853a40d8218"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "004b6249e65c8d56c693c8560301750a"
server_key_iv: "90bd3245920e206cef9a6f968ba6c64c"
server_key_version: "encryptionkey@0"
I20260809 03:33:24.880177 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:24.890173 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:24.893023 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:24.893193 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:24.893705 9147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:24.894544 9147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:24.894591 9147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:24.894641 9147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:24.894668 9147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:24.902622 9147 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:33219
I20260809 03:33:24.902649 9337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:33219 every 8 connection(s)
I20260809 03:33:24.903541 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:33:24.909956 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 9147
I20260809 03:33:24.910348 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:34273
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=45035
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 03:33:24 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246404, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:24.916422 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.905197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:47587 (local address 127.15.222.126:46307)
0809 03:33:24.905320 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:24.905323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:24.906115 (+ 792us) server_negotiation.cc:407] Connection header received
0809 03:33:24.907368 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:24.907371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:24.907431 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:24.907504 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:24.908568 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.909076 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.910283 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.910567 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.910662 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.912901 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:24.912904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:24.912906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:24.912929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:24.914409 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.914887 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.914891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.914893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.915004 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:24.915307 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:24.915310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:24.915312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:24.915553 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:24.915657 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:24.916099 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 03:33:24.916235 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:33:24.917073 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:24.905524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:47587)
0809 03:33:24.905957 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:24.905997 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:24.907181 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:24.907652 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:24.907658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:24.908005 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:24.908457 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.908470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.909186 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.909188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.910154 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:24.910164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:24.910706 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:24.910710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:24.910871 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:24.911393 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:24.911409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:24.912727 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:24.914521 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.914525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.914536 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.914783 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.915090 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:24.915092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:24.915093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:24.915214 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:24.915648 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:24.915652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:24.915868 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 03:33:24.916029 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 03:33:24.916263 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260809 03:33:24.918174 9338 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:24.918401 9338 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:24.918862 9338 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:24.919866 8864 ts_manager.cc:194] Registered new tserver with Master: 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219)
I20260809 03:33:24.920415 8864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:47587
W20260809 03:33:25.023136 9343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:25.023397 9343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:25.023468 9343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:25.027392 9343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:25.027444 9343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:25.027521 9343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:33:25.032557 9343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:34273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=45035
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:25.033522 9343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:25.034538 9343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:25.043749 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:25.043772 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:25.043857 9343 server_base.cc:1034] running on GCE node
W20260809 03:33:25.043807 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:25.044464 9343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:25.045079 9343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:25.046240 9343 hybrid_clock.cc:648] HybridClock initialized: now 1786246405046215 us; error 36 us; skew 500 ppm
Aug 09 03:33:25 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246405, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:25.048893 9343 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:33:25.049751 9343 webserver.cc:460] Webserver started at http://127.15.222.67:45035/ using document root <none> and password file <none>
I20260809 03:33:25.050256 9343 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:25.050325 9343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:25.054109 9343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:33:25.055968 9365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:25.057091 9343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:25.057181 9343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d6ca3f7207274b17b12cf3c516355a91"
format_stamp: "Formatted at 2026-08-09 03:33:04 on dist-test-slave-cvwc"
server_key: "1795a4aa72f5aaa322faff02acc9bf34"
server_key_iv: "5b295ec45cf8a114bdd5ef6ec748a157"
server_key_version: "encryptionkey@0"
I20260809 03:33:25.057786 9343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:25.078914 9343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:25.081655 9343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:25.081833 9343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:25.082369 9343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:25.083139 9343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:25.083184 9343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:25.083222 9343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:25.083242 9343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:25.091727 9343 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34273
I20260809 03:33:25.091760 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34273 every 8 connection(s)
I20260809 03:33:25.092665 9343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:33:25.097815 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 9343
Aug 09 03:33:25 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246405, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:33:25.106083 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.098492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.15.222.126:46307)
0809 03:33:25.098774 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.098778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.098791 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:33:25.098917 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.098921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.098965 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.099035 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:25.099741 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.100248 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.101178 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.101454 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.101553 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.102880 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:25.102884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:25.102887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:25.102927 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:25.104465 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.104875 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.104879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.104880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.104968 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.105212 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.105214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.105215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.105426 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:25.105627 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.105758 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.105876 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260809 03:33:25.106110 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.094365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:35333 (local address 127.15.222.126:46307)
0809 03:33:25.094503 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.094507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.095361 (+ 854us) server_negotiation.cc:407] Connection header received
0809 03:33:25.096624 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.096626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.096672 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.096741 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:25.097928 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.098596 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.099724 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.099937 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.100023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.102503 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:25.102507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:25.102510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:25.102539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:25.103982 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.104538 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.104541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.104543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.104629 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.104969 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.104972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.104973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.105182 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:25.105269 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.105835 (+ 566us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.105943 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 03:33:25.106961 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.094774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:35333)
0809 03:33:25.095207 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.095250 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:25.096434 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:25.096900 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:25.096906 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:25.097237 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:25.097774 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.097791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.098743 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.098745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.099608 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.099618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.100076 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.100080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.100246 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.100769 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:25.100784 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:25.102344 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:25.104096 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:25.104102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:25.104116 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:25.104445 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:25.104717 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:25.104719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:25.104720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:25.104874 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:25.105278 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:25.105284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:25.105563 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 03:33:25.105771 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 03:33:25.106077 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
I20260809 03:33:25.108302 9539 heartbeater.cc:347] Connected to a master server at 127.15.222.126:46307
I20260809 03:33:25.108502 9539 heartbeater.cc:464] Registering TS with master...
I20260809 03:33:25.108915 9539 heartbeater.cc:511] Master 127.15.222.126:46307 requested a full tablet report, sending...
I20260809 03:33:25.109885 8864 ts_manager.cc:194] Registered new tserver with Master: d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
I20260809 03:33:25.110437 8864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:35333
I20260809 03:33:25.119376 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:33:25.122879 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.119934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.15.222.126:46307)
0809 03:33:25.120047 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.120051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.120199 (+ 148us) server_negotiation.cc:407] Connection header received
0809 03:33:25.120317 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.120319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.120403 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.120480 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:33:25.120642 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.121130 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.121953 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.122174 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.122257 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.122466 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:33:25.122578 (+ 112us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 03:33:25.122768 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52468: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 03:33:25.122967 9545 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52468: token expired
W20260809 03:33:25.123238 8742 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.222.126:46307)
I20260809 03:33:25.129235 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.123562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.15.222.126:46307)
0809 03:33:25.123652 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.123655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.123719 (+ 64us) server_negotiation.cc:407] Connection header received
0809 03:33:25.123814 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.123815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.123847 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.123902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:25.124390 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.124825 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.125507 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.125690 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.125801 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.126842 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:25.126849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:25.126851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:25.126871 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:25.128140 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.128440 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.128442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.128443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.128507 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:25.128667 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:25.128668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:25.128669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:25.128801 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:25.128885 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.129020 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.129103 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 03:33:25.131258 8864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:33:25.134305 8864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:33:25.141381 9048 tablet_service.cc:1467] Processing CreateTablet for tablet c307c06ce9af478da01070d11781e6ca (DEFAULT_TABLE table=test-table [id=f6c80aeeda044881b06e885cbb33fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:25.142079 9048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c307c06ce9af478da01070d11781e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:25.144878 8985 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap starting.
I20260809 03:33:25.145030 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.140315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:33219 (local address 127.0.0.1:49042)
0809 03:33:25.140426 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.140440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:25.140509 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:25.141329 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:25.141332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:25.141346 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:25.141438 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.141444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.142802 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.142806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.143564 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.143573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.144606 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.144610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.144701 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.144747 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 03:33:25.144822 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:33:25.144912 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 03:33:25.145494 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.140506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.15.222.66:33219)
0809 03:33:25.140847 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.140859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.140882 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:33:25.140974 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.140979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.141158 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.141580 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:25.141791 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.142703 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.143685 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.144482 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.144586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.144721 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.145300 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.145379 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260809 03:33:25.145780 8985 tablet_bootstrap.cc:654] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:25.147292 8985 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: No bootstrap required, opened a new log
I20260809 03:33:25.147383 8985 ts_tablet_manager.cc:1397] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 03:33:25.147809 8985 raft_consensus.cc:348] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.147938 8985 raft_consensus.cc:374] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:25.147984 8985 raft_consensus.cc:729] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568090aaa92b424d8132cb3f9f9ae561, State: Initialized, Role: FOLLOWER
I20260809 03:33:25.148106 8985 consensus_queue.cc:260] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.148356 8985 ts_tablet_manager.cc:1428] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 03:33:25.148537 9250 tablet_service.cc:1467] Processing CreateTablet for tablet c307c06ce9af478da01070d11781e6ca (DEFAULT_TABLE table=test-table [id=f6c80aeeda044881b06e885cbb33fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:25.149457 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c307c06ce9af478da01070d11781e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:25.154394 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.140986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34273 (local address 127.0.0.1:58982)
0809 03:33:25.141255 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.141267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:25.141343 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:25.142320 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:25.142323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:25.142342 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:25.142448 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.142454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.143758 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.143761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.145428 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.145438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.154065 (+ 8627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.154068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.154142 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.154161 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 03:33:25.154208 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 03:33:25.154263 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260809 03:33:25.154505 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.141411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.15.222.67:34273)
0809 03:33:25.141893 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.141905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.141927 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:33:25.141997 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.142001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.142191 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.142337 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:25.142637 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.143668 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.145562 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.153960 (+ 8398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.154074 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.154210 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.154282 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.154368 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":148,"threads_started":1}
I20260809 03:33:25.155938 9561 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Bootstrap starting.
I20260809 03:33:25.157052 9439 tablet_service.cc:1467] Processing CreateTablet for tablet c307c06ce9af478da01070d11781e6ca (DEFAULT_TABLE table=test-table [id=f6c80aeeda044881b06e885cbb33fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:33:25.158201 9439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c307c06ce9af478da01070d11781e6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:33:25.158869 9561 tablet_bootstrap.cc:654] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:25.159704 9561 log.cc:824] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:25.161806 9561 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: No bootstrap required, opened a new log
I20260809 03:33:25.162009 9561 ts_tablet_manager.cc:1397] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 03:33:25.165191 9561 raft_consensus.cc:348] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.165403 9561 raft_consensus.cc:374] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:25.165450 9561 raft_consensus.cc:729] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff61b057a584467b3c74853a40d8218, State: Initialized, Role: FOLLOWER
I20260809 03:33:25.165877 9561 consensus_queue.cc:260] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.166481 9338 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
I20260809 03:33:25.166754 9561 ts_tablet_manager.cc:1428] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:33:25.167659 9567 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Bootstrap starting.
I20260809 03:33:25.169690 9567 tablet_bootstrap.cc:654] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Neither blocks nor log segments found. Creating new log.
I20260809 03:33:25.170945 9567 log.cc:824] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:25.172612 9567 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: No bootstrap required, opened a new log
I20260809 03:33:25.172781 9567 ts_tablet_manager.cc:1397] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:33:25.175498 9567 raft_consensus.cc:348] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.175714 9567 raft_consensus.cc:374] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:33:25.175751 9567 raft_consensus.cc:729] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ca3f7207274b17b12cf3c516355a91, State: Initialized, Role: FOLLOWER
I20260809 03:33:25.176096 9567 consensus_queue.cc:260] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.176641 9539 heartbeater.cc:503] Master 127.15.222.126:46307 was elected leader, sending a full tablet report...
I20260809 03:33:25.176934 9567 ts_tablet_manager.cc:1428] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260809 03:33:25.219779 9143 tablet.cc:2369] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:33:25.321609 9566 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:25.321847 9566 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.322961 9566 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273), 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435)
I20260809 03:33:25.327037 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.323190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34273 (local address 127.15.222.66:46371)
0809 03:33:25.323325 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.323341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:25.323461 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:25.323747 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:25.323749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:25.323768 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:25.323894 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.323901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.324784 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.324787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.326070 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.326081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.326727 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.326729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.326805 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.326826 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:33:25.326883 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 03:33:25.326937 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":46}
I20260809 03:33:25.327097 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.323357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46371 (local address 127.15.222.67:34273)
0809 03:33:25.323508 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.323512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.323530 (+ 18us) server_negotiation.cc:407] Connection header received
0809 03:33:25.323590 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.323592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.323644 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.323728 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:25.324019 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.324682 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.326185 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.326618 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.326734 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.326785 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.326911 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.326974 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 03:33:25.327675 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.323190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:34435 (local address 127.15.222.66:40291)
0809 03:33:25.323994 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.324007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:25.324105 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:25.324353 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:25.324355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:25.324372 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:33:25.324483 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.324489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.325396 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.325399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.326471 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:25.326479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.327343 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.327347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:25.327448 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.327472 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:33:25.327527 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 03:33:25.327576 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":402,"thread_start_us":139,"threads_started":1}
I20260809 03:33:25.327736 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.323375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40291 (local address 127.15.222.65:34435)
0809 03:33:25.323667 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.323670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.324072 (+ 402us) server_negotiation.cc:407] Connection header received
0809 03:33:25.324181 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.324183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.324237 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.324322 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:33:25.324577 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.325268 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.326588 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.327211 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.327333 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.327494 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.327541 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.327614 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260809 03:33:25.327688 9466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c307c06ce9af478da01070d11781e6ca" candidate_uuid: "5ff61b057a584467b3c74853a40d8218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ca3f7207274b17b12cf3c516355a91" is_pre_election: true
I20260809 03:33:25.328015 9466 raft_consensus.cc:2454] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff61b057a584467b3c74853a40d8218 in term 0.
I20260809 03:33:25.328326 9075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c307c06ce9af478da01070d11781e6ca" candidate_uuid: "5ff61b057a584467b3c74853a40d8218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568090aaa92b424d8132cb3f9f9ae561" is_pre_election: true
I20260809 03:33:25.328507 9181 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91; no voters:
I20260809 03:33:25.328584 9075 raft_consensus.cc:2454] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff61b057a584467b3c74853a40d8218 in term 0.
I20260809 03:33:25.328774 9566 raft_consensus.cc:2781] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:33:25.328859 9566 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:33:25.328907 9566 raft_consensus.cc:3037] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:25.329959 9566 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.330277 9566 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [CANDIDATE]: Term 1 election: Requested vote from peers d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273), 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435)
I20260809 03:33:25.330600 9466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c307c06ce9af478da01070d11781e6ca" candidate_uuid: "5ff61b057a584467b3c74853a40d8218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ca3f7207274b17b12cf3c516355a91"
I20260809 03:33:25.330641 9075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c307c06ce9af478da01070d11781e6ca" candidate_uuid: "5ff61b057a584467b3c74853a40d8218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568090aaa92b424d8132cb3f9f9ae561"
I20260809 03:33:25.330703 9466 raft_consensus.cc:3037] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:25.330754 9075 raft_consensus.cc:3037] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:33:25.331679 9466 raft_consensus.cc:2454] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff61b057a584467b3c74853a40d8218 in term 1.
I20260809 03:33:25.331689 9075 raft_consensus.cc:2454] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff61b057a584467b3c74853a40d8218 in term 1.
I20260809 03:33:25.331946 9182 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561, 5ff61b057a584467b3c74853a40d8218; no voters:
I20260809 03:33:25.332099 9566 raft_consensus.cc:2781] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:33:25.332304 9566 raft_consensus.cc:686] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff61b057a584467b3c74853a40d8218, State: Running, Role: LEADER
I20260809 03:33:25.332594 9566 consensus_queue.cc:237] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:25.335589 8864 catalog_manager.cc:5495] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff61b057a584467b3c74853a40d8218 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "5ff61b057a584467b3c74853a40d8218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } health_report { overall_health: UNKNOWN } } }
W20260809 03:33:25.343473 9541 tablet.cc:2369] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:33:25.379166 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.375848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.15.222.66:33219)
0809 03:33:25.375996 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:25.376000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:25.376011 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:33:25.376079 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:25.376081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:25.376132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:25.376219 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:33:25.376463 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.376953 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.377756 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:25.377981 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:25.378067 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:25.378340 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:33:25.378801 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:33:25.378916 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:25.378975 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 03:33:25.379031 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 03:33:25.384975 9466 raft_consensus.cc:1261] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Refusing update from remote peer 5ff61b057a584467b3c74853a40d8218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:33:25.385002 9075 raft_consensus.cc:1261] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Refusing update from remote peer 5ff61b057a584467b3c74853a40d8218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:33:25.385663 9566 consensus_queue.cc:1035] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:33:25.385874 9573 consensus_queue.cc:1035] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:33:25.399261 8862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:33:25.399444 8862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:33:25.401682 8862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Sending DeleteTablet for 3 replicas of tablet c307c06ce9af478da01070d11781e6ca
I20260809 03:33:25.402398 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet c307c06ce9af478da01070d11781e6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:49042
I20260809 03:33:25.402438 9048 tablet_service.cc:1514] Processing DeleteTablet for tablet c307c06ce9af478da01070d11781e6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260809 03:33:25.402501 9439 tablet_service.cc:1514] Processing DeleteTablet for tablet c307c06ce9af478da01070d11781e6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:33:25 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58982
I20260809 03:33:25.402760 9589 tablet_replica.cc:330] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: stopping tablet replica
I20260809 03:33:25.402871 9589 raft_consensus.cc:2229] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:33:25.403033 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8917
I20260809 03:33:25.403119 9591 tablet_replica.cc:330] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: stopping tablet replica
I20260809 03:33:25.403144 9589 raft_consensus.cc:2258] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:25.403430 9591 raft_consensus.cc:2229] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:33:25.403774 9591 raft_consensus.cc:2258] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:25.404210 9590 tablet_replica.cc:330] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: stopping tablet replica
I20260809 03:33:25.408526 9591 ts_tablet_manager.cc:1905] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:33:25.411222 9591 ts_tablet_manager.cc:1918] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:33:25.411316 9591 log.cc:1196] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/c307c06ce9af478da01070d11781e6ca
I20260809 03:33:25.411643 9591 ts_tablet_manager.cc:1939] T c307c06ce9af478da01070d11781e6ca P d6ca3f7207274b17b12cf3c516355a91: Deleting consensus metadata
I20260809 03:33:25.412846 8840 catalog_manager.cc:4841] TS d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): tablet c307c06ce9af478da01070d11781e6ca (table test-table [id=f6c80aeeda044881b06e885cbb33fc2a]) successfully deleted
I20260809 03:33:25.413849 9590 raft_consensus.cc:2229] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:33:25.414175 9590 raft_consensus.cc:2258] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:33:25.415614 9590 ts_tablet_manager.cc:1905] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:33:25.418157 9590 ts_tablet_manager.cc:1918] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:33:25.418259 9590 log.cc:1196] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/c307c06ce9af478da01070d11781e6ca
I20260809 03:33:25.418592 9590 ts_tablet_manager.cc:1939] T c307c06ce9af478da01070d11781e6ca P 5ff61b057a584467b3c74853a40d8218: Deleting consensus metadata
I20260809 03:33:25.419683 8840 catalog_manager.cc:4841] TS 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): tablet c307c06ce9af478da01070d11781e6ca (table test-table [id=f6c80aeeda044881b06e885cbb33fc2a]) successfully deleted
W20260809 03:33:25.419925 8842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:34435 (error 108)
I20260809 03:33:25.420037 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 9147
I20260809 03:33:25.430999 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:25.430692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:34435 (local address 127.0.0.1:33150)
0809 03:33:25.430857 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:25.430922 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260809 03:33:25.431138 8842 catalog_manager.cc:4584] TS 568090aaa92b424d8132cb3f9f9ae561 (127.15.222.65:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c307c06ce9af478da01070d11781e6ca: Network error: Client connection negotiation failed: client connection to 127.15.222.65:34435: connect: Connection refused (error 111)
I20260809 03:33:25.437211 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 9343
I20260809 03:33:25.459649 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 8806
I20260809 03:33:45.469146 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--webserver_interface=127.15.222.126
--webserver_port=36987
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:33:45.578258 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:45.578538 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:45.578595 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:45.582499 9597 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 03:33:45.582549 9597 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 03:33:45.582571 9597 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 03:33:45.582587 9597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:33:45.582612 9597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:45.582645 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:45.582669 9597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:33:45.582697 9597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:33:45.587738 9597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:45.588708 9597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:45.589684 9597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:45.597875 9607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:45.597985 9597 server_base.cc:1034] running on GCE node
W20260809 03:33:45.597935 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:45.597914 9609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:45.598695 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:45.599963 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:45.601125 9597 hybrid_clock.cc:648] HybridClock initialized: now 1786246425601111 us; error 28 us; skew 500 ppm
Aug 09 03:33:45 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:45.603750 9597 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:33:45.604568 9597 webserver.cc:460] Webserver started at http://127.15.222.126:36987/ using document root <none> and password file <none>
I20260809 03:33:45.605093 9597 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:45.605160 9597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:45.608793 9597 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:45.610850 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:45.611959 9597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:33:45.612071 9597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "7ac9e0ec4c584a9bb92065adae1e8274"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "0c720e3568b67107940358dd039871f2"
server_key_iv: "4f18a5c158c0f8938df495f0ccca12c2"
server_key_version: "encryptionkey@0"
I20260809 03:33:45.612584 9597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:45.628975 9597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:45.632153 9597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:45.632324 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:45.639285 9597 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46307
I20260809 03:33:45.639560 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46307 every 8 connection(s)
I20260809 03:33:45.640339 9597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:33:45.644635 9694 sys_catalog.cc:263] Verifying existing consensus state
I20260809 03:33:45.645910 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 9597
I20260809 03:33:45.646453 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:34435
--local_ip_for_outbound_sockets=127.15.222.65
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=36101
--webserver_interface=127.15.222.65
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:33:45.646495 9694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap starting.
I20260809 03:33:45.664062 9694 log.cc:824] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:45.676191 9694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:33:45.676559 9694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Bootstrap complete.
I20260809 03:33:45.680471 9694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:45.680877 9694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Initialized, Role: FOLLOWER
I20260809 03:33:45.681409 9694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:45.681555 9694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:33:45.681629 9694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:33:45.681754 9694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 2 FOLLOWER]: Advancing to term 3
I20260809 03:33:45.683655 9694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:45.683988 9694 leader_election.cc:302] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac9e0ec4c584a9bb92065adae1e8274; no voters:
I20260809 03:33:45.684321 9694 leader_election.cc:288] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 03:33:45.684430 9701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 03:33:45.684623 9701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [term 3 LEADER]: Becoming Leader. State: Replica: 7ac9e0ec4c584a9bb92065adae1e8274, State: Running, Role: LEADER
I20260809 03:33:45.685009 9701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } }
I20260809 03:33:45.685602 9694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:33:45.686525 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac9e0ec4c584a9bb92065adae1e8274. Latest consensus state: current_term: 3 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:45.686571 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac9e0ec4c584a9bb92065adae1e8274" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46307 } } }
I20260809 03:33:45.686699 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:45.686647 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274 [sys.catalog]: This master's current role is: LEADER
I20260809 03:33:45.687139 9716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:33:45.692538 9716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:33:45.693404 9716 catalog_manager.cc:1259] Loaded cluster ID: 91ee91af01b046c2a52883cc0962f75e
I20260809 03:33:45.693480 9716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:33:45.696126 9716 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:33:45.696923 9716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Loaded TSK: 0
I20260809 03:33:45.697013 9716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ac9e0ec4c584a9bb92065adae1e8274: Loaded TSK: 1
F20260809 03:33:45.697872 9716 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a64a309ad google::LogMessage::Fail() at ??:0
@ 0x7f5a64a3388a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a64a304de google::LogMessage::Flush() at ??:0
@ 0x7f5a64a31a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a676f418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5a676ecf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5a6f3997bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5a6f37399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5a6f374178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f5a6f3b5df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5a6f3d482b std::function<>::operator()() at ??:0
@ 0x7f5a6f373b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5a6f374f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a6f371579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5a6f3b5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5a66c823d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a66c82c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5a66c843f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5a66c73512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a682b76db start_thread at ??:0
@ 0x7f5a6352e61f clone at ??:0
W20260809 03:33:45.774647 9698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:45.774947 9698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:45.774995 9698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:45.779165 9698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:45.779208 9698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:45.779264 9698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:33:45.784688 9698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=36101
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:45.785758 9698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:45.786717 9698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:45.796104 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:45.796249 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:45.796119 9764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:45.796514 9698 server_base.cc:1034] running on GCE node
I20260809 03:33:45.796991 9698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:45.797622 9698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:45.798813 9698 hybrid_clock.cc:648] HybridClock initialized: now 1786246425798784 us; error 45 us; skew 500 ppm
Aug 09 03:33:45 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:45.801471 9698 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:33:45.802403 9698 webserver.cc:460] Webserver started at http://127.15.222.65:36101/ using document root <none> and password file <none>
I20260809 03:33:45.802960 9698 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:45.803036 9698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:45.806866 9698 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 03:33:45.808943 9784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:45.810212 9698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:33:45.810324 9698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "568090aaa92b424d8132cb3f9f9ae561"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "edfe0e468fdc505438d7f23501467d98"
server_key_iv: "e8d5f9190f1a22eb798cf57d2c5e0b00"
server_key_version: "encryptionkey@0"
I20260809 03:33:45.810891 9698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:45.830127 9698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:45.833231 9698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:45.833403 9698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:45.834061 9698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:45.835142 9797 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 03:33:45.839337 9698 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 03:33:45.839392 9698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 03:33:45.839442 9698 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 03:33:45.841042 9698 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 03:33:45.841097 9698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 03:33:45.841374 9797 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap starting.
I20260809 03:33:45.851203 9698 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:34435
I20260809 03:33:45.851436 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:34435 every 8 connection(s)
I20260809 03:33:45.852380 9698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:33:45.853955 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 9698
I20260809 03:33:45.854432 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:33219
--local_ip_for_outbound_sockets=127.15.222.66
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=36463
--webserver_interface=127.15.222.66
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:33:45.864074 9797 log.cc:824] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Log is configured to *not* fsync() on all Append() calls
I20260809 03:33:45.873291 9797 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 03:33:45.873744 9797 tablet_bootstrap.cc:492] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Bootstrap complete.
I20260809 03:33:45.875068 9797 ts_tablet_manager.cc:1397] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.013s
I20260809 03:33:45.878257 9797 raft_consensus.cc:348] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:45.878803 9797 raft_consensus.cc:729] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568090aaa92b424d8132cb3f9f9ae561, State: Initialized, Role: FOLLOWER
I20260809 03:33:45.879365 9797 consensus_queue.cc:260] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:45.880355 9797 ts_tablet_manager.cc:1428] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260809 03:33:45.934332 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:33:45.857608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:55577)
0809 03:33:45.862049 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:45.862111 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:45.863119 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:45.933329 (+ 70210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4180,"thread_start_us":135,"threads_started":1}
W20260809 03:33:45.934986 9956 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46307 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46307: Client connection negotiation failed: client connection to 127.15.222.126:46307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 03:33:45.935742 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:45.935456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:48413)
0809 03:33:45.935599 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:45.935677 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 03:33:45.936486 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:45.936260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:58061)
0809 03:33:45.936372 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:45.936431 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 03:33:45.936651 9956 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:33:45.979581 9961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:45.979812 9961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:45.979851 9961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:45.983686 9961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:45.983736 9961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:45.983789 9961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:33:45.988765 9961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:33219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=36463
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:45.989691 9961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:45.990644 9961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:45.999727 9975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:45.999787 9971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:45.999758 9972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:46.000061 9961 server_base.cc:1034] running on GCE node
I20260809 03:33:46.000439 9961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:46.001025 9961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:46.002173 9961 hybrid_clock.cc:648] HybridClock initialized: now 1786246426002159 us; error 24 us; skew 500 ppm
Aug 09 03:33:45 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:46.004731 9961 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:33:46.005574 9961 webserver.cc:460] Webserver started at http://127.15.222.66:36463/ using document root <none> and password file <none>
I20260809 03:33:46.006126 9961 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:46.006188 9961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:46.009757 9961 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 03:33:46.011616 9983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.012737 9961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:33:46.012833 9961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "5ff61b057a584467b3c74853a40d8218"
format_stamp: "Formatted at 2026-08-09 03:33:03 on dist-test-slave-cvwc"
server_key: "004b6249e65c8d56c693c8560301750a"
server_key_iv: "90bd3245920e206cef9a6f968ba6c64c"
server_key_version: "encryptionkey@0"
I20260809 03:33:46.013337 9961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:46.027462 9961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:46.029944 9961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:46.030081 9961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:46.030563 9961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:46.031306 9961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:46.031349 9961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.031384 9961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:46.031404 9961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.039808 9961 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:33219
I20260809 03:33:46.039834 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:33219 every 8 connection(s)
I20260809 03:33:46.041118 9961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:33:46.041448 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 9961
I20260809 03:33:46.041950 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:34273
--local_ip_for_outbound_sockets=127.15.222.67
--tserver_master_addrs=127.15.222.126:46307
--webserver_port=45035
--webserver_interface=127.15.222.67
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:33:46.045522 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.043387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:58613)
0809 03:33:46.043927 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.044089 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
W20260809 03:33:46.045974 10157 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46307 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46307: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
I20260809 03:33:46.046572 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.046370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:33189)
0809 03:33:46.046473 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.046518 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 03:33:46.047173 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.046983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:56037)
0809 03:33:46.047068 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.047112 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 03:33:46.047369 10157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 03:33:46.152361 10160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:33:46.152582 10160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:33:46.152624 10160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:33:46.156494 10160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:33:46.156544 10160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:33:46.156610 10160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:33:46.161490 10160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:34273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=45035
--tserver_master_addrs=127.15.222.126:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:33:46.162410 10160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:33:46.163409 10160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:33:46.172446 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:46.172456 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:33:46.172505 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:33:46.172667 10160 server_base.cc:1034] running on GCE node
I20260809 03:33:46.173050 10160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:33:46.173625 10160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:33:46.174767 10160 hybrid_clock.cc:648] HybridClock initialized: now 1786246426174749 us; error 29 us; skew 500 ppm
Aug 09 03:33:46 dist-test-slave-cvwc krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246426, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:33:46.177245 10160 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:33:46.178107 10160 webserver.cc:460] Webserver started at http://127.15.222.67:45035/ using document root <none> and password file <none>
I20260809 03:33:46.178663 10160 fs_manager.cc:359] Metadata directory not provided
I20260809 03:33:46.178745 10160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:33:46.182298 10160 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:33:46.184175 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.185303 10160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:33:46.185418 10160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "d6ca3f7207274b17b12cf3c516355a91"
format_stamp: "Formatted at 2026-08-09 03:33:04 on dist-test-slave-cvwc"
server_key: "1795a4aa72f5aaa322faff02acc9bf34"
server_key_iv: "5b295ec45cf8a114bdd5ef6ec748a157"
server_key_version: "encryptionkey@0"
I20260809 03:33:46.185961 10160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:33:46.201120 10160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:33:46.203666 10160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:33:46.203822 10160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:33:46.204357 10160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:33:46.205128 10160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:33:46.205173 10160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.205215 10160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:33:46.205243 10160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:33:46.213406 10160 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34273
I20260809 03:33:46.213439 10344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34273 every 8 connection(s)
I20260809 03:33:46.214465 10160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:33:46.218201 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.216439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:57885)
0809 03:33:46.216851 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.217044 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
W20260809 03:33:46.218619 10345 heartbeater.cc:644] Failed to heartbeat to 127.15.222.126:46307 (0 consecutive failures): Network error: Failed to ping master at 127.15.222.126:46307: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
I20260809 03:33:46.219158 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.218963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:37959)
0809 03:33:46.219060 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.219106 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 03:33:46.219446 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 10160
I20260809 03:33:46.219925 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.219542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:59523)
0809 03:33:46.219805 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.219855 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
W20260809 03:33:46.220088 10345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 03:33:46.937618 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:46.937179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:57747)
0809 03:33:46.937461 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:46.937515 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260809 03:33:47.013356 10353 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:47.013562 10353 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:47.014701 10353 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219), d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
Aug 09 03:33:47 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.67@KRBTEST.COM
Aug 09 03:33:47 dist-test-slave-cvwc krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.66@KRBTEST.COM
I20260809 03:33:47.027382 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.014884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:33219 (local address 127.15.222.65:57491)
0809 03:33:47.015021 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:47.015036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:47.015152 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:47.017078 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:47.017084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:47.017683 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:47.018227 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:47.018243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.019487 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.019490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:47.020376 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:47.020394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.021036 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.021040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:47.021232 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:47.021936 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:47.021948 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:47.023697 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:47.025529 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:47.025532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:47.025535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:47.025740 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:47.026041 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:47.026045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:47.026046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:47.026120 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:47.026651 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:47.026654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:47.026744 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 03:33:47.026864 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 03:33:47.027196 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":121,"spinlock_wait_cycles":38272}
I20260809 03:33:47.027446 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.015346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:52511 (local address 127.15.222.67:34273)
0809 03:33:47.015625 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:47.015637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:47.015685 (+ 48us) server_negotiation.cc:407] Connection header received
0809 03:33:47.016290 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:47.016297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:47.016751 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:47.017054 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:47.018386 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.019355 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.020487 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.020921 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.021066 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:47.023561 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:47.023566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:47.023577 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:47.023603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:47.025104 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:47.025566 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:47.025571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:47.025575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:47.025683 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:47.026027 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:47.026030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:47.026032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:47.026389 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:47.026521 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:47.027009 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 03:33:47.027254 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260809 03:33:47.027415 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.015078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:57491 (local address 127.15.222.66:33219)
0809 03:33:47.015459 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0809 03:33:47.015476 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 03:33:47.015525 (+ 49us) server_negotiation.cc:407] Connection header received
0809 03:33:47.016237 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:33:47.016243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:33:47.016708 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:33:47.017025 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:33:47.018383 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.019350 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.020492 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.020929 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.021068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:47.023862 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:33:47.023868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:33:47.023882 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:33:47.023913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:33:47.025427 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:47.025838 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:47.025843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:47.025849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:47.025954 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:33:47.026207 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:33:47.026210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:33:47.026212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:33:47.026551 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:33:47.026662 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 03:33:47.027006 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 03:33:47.027251 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260809 03:33:47.027434 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.015147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34273 (local address 127.15.222.65:52511)
0809 03:33:47.015508 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:47.015521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:33:47.015620 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:33:47.017334 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:33:47.017337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:33:47.017647 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:33:47.018228 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:47.018243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.019480 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.019483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:47.020378 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:33:47.020387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:33:47.021030 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:33:47.021033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:33:47.021227 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:33:47.021902 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:33:47.021932 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:33:47.023361 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:33:47.025215 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:47.025220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:47.025233 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:47.025463 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:47.025796 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:33:47.025799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:33:47.025801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:33:47.025942 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:33:47.026515 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:33:47.026520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:33:47.026720 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 03:33:47.026864 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 03:33:47.027269 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":244,"spinlock_wait_cycles":2816,"thread_start_us":68,"threads_started":1}
W20260809 03:33:47.030102 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
W20260809 03:33:47.030288 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
I20260809 03:33:47.030347 9787 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561; no voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91
I20260809 03:33:47.030611 10353 raft_consensus.cc:2726] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:47.048105 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.047784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:40105)
0809 03:33:47.047978 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:47.048023 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260809 03:33:47.220885 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.220537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:58561)
0809 03:33:47.220754 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:47.220803 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260809 03:33:47.938707 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:47.938312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:35559)
0809 03:33:47.938554 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:47.938611 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260809 03:33:48.049144 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:48.048736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:60503)
0809 03:33:48.048974 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:48.049027 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260809 03:33:48.221879 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:48.221477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:46899)
0809 03:33:48.221697 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:48.221775 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260809 03:33:48.724036 10368 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:48.724169 10368 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:48.724535 10368 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219), d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
W20260809 03:33:48.725307 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
W20260809 03:33:48.725471 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
I20260809 03:33:48.725533 9787 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561; no voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91
I20260809 03:33:48.725692 10368 raft_consensus.cc:2726] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:48.941352 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:48.940083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:54333)
0809 03:33:48.940826 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:48.941043 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":255,"threads_started":1}
I20260809 03:33:49.052167 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:49.050845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:43171)
0809 03:33:49.051652 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:49.051850 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":225,"threads_started":1}
I20260809 03:33:49.222924 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:49.222547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:55883)
0809 03:33:49.222789 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:49.222841 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260809 03:33:49.943091 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:49.942675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:42513)
0809 03:33:49.942931 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:49.942990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260809 03:33:50.053872 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:50.053427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:53095)
0809 03:33:50.053686 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:50.053770 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260809 03:33:50.224004 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:50.223616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:40693)
0809 03:33:50.223843 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:50.223913 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260809 03:33:50.944324 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:50.943920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:39733)
0809 03:33:50.944175 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:50.944228 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260809 03:33:51.054965 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:51.054541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:41027)
0809 03:33:51.054803 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:51.054859 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260809 03:33:51.085433 10388 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:51.085563 10388 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:51.085978 10388 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219), d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
W20260809 03:33:51.086747 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
W20260809 03:33:51.086864 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
I20260809 03:33:51.086927 9787 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561; no voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91
I20260809 03:33:51.087071 10388 raft_consensus.cc:2726] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:51.225127 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:51.224717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:40933)
0809 03:33:51.224989 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:51.225039 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
I20260809 03:33:51.945341 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:51.944957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:40981)
0809 03:33:51.945194 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:51.945246 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260809 03:33:52.055991 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:52.055589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:42473)
0809 03:33:52.055841 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:52.055900 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260809 03:33:52.226186 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:52.225802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:60011)
0809 03:33:52.226045 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:52.226097 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260809 03:33:52.946491 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:52.946088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:60051)
0809 03:33:52.946332 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:52.946383 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260809 03:33:53.057092 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:53.056652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:51085)
0809 03:33:53.056942 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:53.057001 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260809 03:33:53.227219 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:53.226799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:59321)
0809 03:33:53.227080 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:53.227131 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260809 03:33:53.947547 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:53.947151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:46477)
0809 03:33:53.947399 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:53.947452 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260809 03:33:54.058128 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:54.057759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:36365)
0809 03:33:54.057986 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:54.058041 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260809 03:33:54.228193 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:54.227805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:51617)
0809 03:33:54.228054 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:54.228108 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260809 03:33:54.948683 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:54.948255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:44375)
0809 03:33:54.948523 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:54.948579 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260809 03:33:55.059206 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:55.058769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:53865)
0809 03:33:55.059054 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:55.059112 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260809 03:33:55.229220 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:55.228832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:54561)
0809 03:33:55.229076 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:55.229129 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260809 03:33:55.465488 10406 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:55.465634 10406 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:55.466054 10406 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219), d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
W20260809 03:33:55.466804 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
W20260809 03:33:55.466921 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
I20260809 03:33:55.466971 9787 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561; no voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91
I20260809 03:33:55.467124 10406 raft_consensus.cc:2726] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:55.949980 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:55.949521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:36507)
0809 03:33:55.949791 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:55.949850 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260809 03:33:56.060382 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:56.059989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:60005)
0809 03:33:56.060235 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:56.060290 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260809 03:33:56.230259 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:56.229906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:55607)
0809 03:33:56.230118 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:56.230170 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260809 03:33:56.951118 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:56.950673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:43921)
0809 03:33:56.950962 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:56.951019 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":114,"threads_started":1}
I20260809 03:33:57.054153 10411 raft_consensus.cc:482] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:33:57.054275 10411 raft_consensus.cc:504] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff61b057a584467b3c74853a40d8218" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33219 } } peers { permanent_uuid: "d6ca3f7207274b17b12cf3c516355a91" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34273 } } peers { permanent_uuid: "568090aaa92b424d8132cb3f9f9ae561" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 34435 } }
I20260809 03:33:57.054646 10411 leader_election.cc:288] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219), d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273)
W20260809 03:33:57.055397 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6ca3f7207274b17b12cf3c516355a91 (127.15.222.67:34273): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
W20260809 03:33:57.055558 9787 leader_election.cc:341] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ff61b057a584467b3c74853a40d8218 (127.15.222.66:33219): Not found: Tablet not found: c307c06ce9af478da01070d11781e6ca
I20260809 03:33:57.055610 9787 leader_election.cc:302] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 568090aaa92b424d8132cb3f9f9ae561; no voters: 5ff61b057a584467b3c74853a40d8218, d6ca3f7207274b17b12cf3c516355a91
I20260809 03:33:57.055768 10411 raft_consensus.cc:2726] T c307c06ce9af478da01070d11781e6ca P 568090aaa92b424d8132cb3f9f9ae561 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 03:33:57.061353 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:57.061004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:43881)
0809 03:33:57.061224 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:57.061271 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260809 03:33:57.231241 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:57.230855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:56207)
0809 03:33:57.231108 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:57.231158 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260809 03:33:57.952152 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:57.951779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:41053)
0809 03:33:57.951999 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:57.952054 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260809 03:33:58.062397 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:58.062025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:35775)
0809 03:33:58.062251 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:58.062304 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260809 03:33:58.232201 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:58.231843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:35085)
0809 03:33:58.232059 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:58.232110 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260809 03:33:58.953200 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:58.952829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:32781)
0809 03:33:58.953050 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:58.953101 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260809 03:33:59.063433 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:59.063061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:59921)
0809 03:33:59.063273 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:59.063326 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260809 03:33:59.233304 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:59.232848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:35775)
0809 03:33:59.233133 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:59.233201 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":99,"threads_started":1}
I20260809 03:33:59.954304 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:33:59.953904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:47739)
0809 03:33:59.954152 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 03:33:59.954206 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260809 03:34:00.064538 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:00.064135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:49037)
0809 03:34:00.064387 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:00.064444 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260809 03:34:00.234360 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:00.233959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:36961)
0809 03:34:00.234214 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:00.234267 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260809 03:34:00.955631 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:00.955203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.65:54961)
0809 03:34:00.955476 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:00.955533 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260809 03:34:01.065755 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:01.065313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.66:53137)
0809 03:34:01.065581 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:01.065636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260809 03:34:01.221829 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 9698
I20260809 03:34:01.233006 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 9961
I20260809 03:34:01.240860 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:01.240471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:46307 (local address 127.15.222.67:38901)
0809 03:34:01.240709 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:01.240769 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.126:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260809 03:34:01.244937 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 10160
2026-08-09T03:34:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59279 ms)
[----------] 30 tests from SecurityITest (240673 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:01 dist-test-slave-cvwc krb5kdc[10439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:01 dist-test-slave-cvwc krb5kdc[10439](info): set up 2 sockets
Aug 09 03:34:01 dist-test-slave-cvwc krb5kdc[10439](info): commencing operation
krb5kdc: starting...
W20260809 03:34:02.392838 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:02 dist-test-slave-cvwc krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:02.508037 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38859
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:02.619637 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:02.619901 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:02.619947 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:02.623899 10457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:02.623958 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:02.623976 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:02.623991 10457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:02.624006 10457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:02.629114 10457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:02.630103 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:02.631074 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:02.639460 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:02.639421 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:02.639436 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:02.639508 10457 server_base.cc:1034] running on GCE node
I20260809 03:34:02.640158 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:02.641191 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:02.642338 10457 hybrid_clock.cc:648] HybridClock initialized: now 1786246442642322 us; error 28 us; skew 500 ppm
Aug 09 03:34:02 dist-test-slave-cvwc krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:02.644920 10457 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:02.645781 10457 webserver.cc:460] Webserver started at http://127.15.222.126:37653/ using document root <none> and password file <none>
I20260809 03:34:02.646306 10457 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:02.646365 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:02.646582 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:02.648586 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "810fe1cbbbba4194b2e418fc8b57f2c1"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "d3fb6e3da87f7d19250da97e49de1bd7"
server_key_iv: "70a0be2411bf397d4771fc28e6811853"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.649117 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "810fe1cbbbba4194b2e418fc8b57f2c1"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "d3fb6e3da87f7d19250da97e49de1bd7"
server_key_iv: "70a0be2411bf397d4771fc28e6811853"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.652657 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:02.654932 10483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:02.656026 10457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:02.656131 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "810fe1cbbbba4194b2e418fc8b57f2c1"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "d3fb6e3da87f7d19250da97e49de1bd7"
server_key_iv: "70a0be2411bf397d4771fc28e6811853"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.656227 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:02.674149 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:02.676895 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:02.677074 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:02.684526 10457 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38859
I20260809 03:34:02.684544 10558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38859 every 8 connection(s)
I20260809 03:34:02.685483 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:02.688230 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:02.693493 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 10457
I20260809 03:34:02.693602 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:02.693778 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: Bootstrap starting.
I20260809 03:34:02.693900 16249 external_mini_cluster.cc:1428] Setting key f9d14417825557330f27835463f431fd
I20260809 03:34:02.696182 10562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:02.696875 10562 log.cc:824] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:02.698894 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: No bootstrap required, opened a new log
Aug 09 03:34:02 dist-test-slave-cvwc krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:02.703053 10562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } }
I20260809 03:34:02.703332 10562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:02.703397 10562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810fe1cbbbba4194b2e418fc8b57f2c1, State: Initialized, Role: FOLLOWER
I20260809 03:34:02.703800 10562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } }
I20260809 03:34:02.703936 10562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:02.704001 10562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:02.704100 10562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:02.704911 10562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } }
I20260809 03:34:02.705224 10562 leader_election.cc:302] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 810fe1cbbbba4194b2e418fc8b57f2c1; no voters:
I20260809 03:34:02.705540 10562 leader_election.cc:288] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:02.705619 10567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:02.705811 10567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [term 1 LEADER]: Becoming Leader. State: Replica: 810fe1cbbbba4194b2e418fc8b57f2c1, State: Running, Role: LEADER
I20260809 03:34:02.706169 10567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } }
I20260809 03:34:02.706848 10562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:02.707405 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 810fe1cbbbba4194b2e418fc8b57f2c1. Latest consensus state: current_term: 1 leader_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } } }
I20260809 03:34:02.707515 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:02.708137 10568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fe1cbbbba4194b2e418fc8b57f2c1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38859 } } }
I20260809 03:34:02.708233 10568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:02.708510 10574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:02.708599 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:02.695232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.15.222.126:38859)
0809 03:34:02.695645 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:02.695661 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:02.695696 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:34:02.696344 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:02.696363 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:02.696684 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:02.696957 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:02.697802 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.698860 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.700002 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.700478 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.700591 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:02.703416 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:02.703422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:02.703441 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:02.703472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:02.705247 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:02.705698 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:02.705705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:02.705740 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:02.705892 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:02.706206 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:02.706210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:02.706211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:02.706636 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:02.706894 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:02.707451 (+ 557us) server_negotiation.cc:299] Negotiation successful
0809 03:34:02.707747 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260809 03:34:02.711632 10574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:02.717463 10574 catalog_manager.cc:1347] Generated new cluster ID: d08891d3765d450b9b929f0ceb2888a2
I20260809 03:34:02.717520 10574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:02.732110 10574 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:02.733062 10574 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:02.736749 10574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: Generated new TSK 0
I20260809 03:34:02.737530 10574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:34:02.772595 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:02.893509 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:02.893774 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:02.893818 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:02.897737 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:02.897784 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:02.897855 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:02.903031 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:02.903998 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:02.904994 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:02.914485 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:02.914570 10604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:02.914824 10597 server_base.cc:1034] running on GCE node
W20260809 03:34:02.914499 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:02.915375 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:02.916126 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:02.917287 10597 hybrid_clock.cc:648] HybridClock initialized: now 1786246442917250 us; error 51 us; skew 500 ppm
Aug 09 03:34:02 dist-test-slave-cvwc krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:02.919899 10597 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:34:02.920714 10597 webserver.cc:460] Webserver started at http://127.15.222.65:35063/ using document root <none> and password file <none>
I20260809 03:34:02.921240 10597 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:02.921304 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:02.921501 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:02.923535 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "81b59ebf8e674fc698a4c27fd26b4d7b"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "e151d0cf5778c739f5d003f086835f2a"
server_key_iv: "9cb4974a32391127618a907a04e70590"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.924101 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "81b59ebf8e674fc698a4c27fd26b4d7b"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "e151d0cf5778c739f5d003f086835f2a"
server_key_iv: "9cb4974a32391127618a907a04e70590"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.927714 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:02.930219 10616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:02.931365 10597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:02.931481 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "81b59ebf8e674fc698a4c27fd26b4d7b"
format_stamp: "Formatted at 2026-08-09 03:34:02 on dist-test-slave-cvwc"
server_key: "e151d0cf5778c739f5d003f086835f2a"
server_key_iv: "9cb4974a32391127618a907a04e70590"
server_key_version: "encryptionkey@0"
I20260809 03:34:02.931571 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:02.940044 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:02.942756 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:02.942922 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:02.943440 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:02.944217 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:02.944262 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:02.944314 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:02.944339 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:02.952379 10597 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44757
I20260809 03:34:02.952416 10782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44757 every 8 connection(s)
I20260809 03:34:02.953213 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:02.960071 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 10597
I20260809 03:34:02.960196 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:34:02.960580 16249 external_mini_cluster.cc:1428] Setting key cb7bfae57d52ed13dffa29daaca97500
Aug 09 03:34:02 dist-test-slave-cvwc krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:02.966418 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:02.954936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:54175 (local address 127.15.222.126:38859)
0809 03:34:02.955062 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:02.955066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:02.955827 (+ 761us) server_negotiation.cc:407] Connection header received
0809 03:34:02.957037 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:02.957040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:02.957086 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:02.957156 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:02.958264 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.958792 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.959915 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.960281 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.960389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:02.963033 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:02.963036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:02.963038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:02.963060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:02.964453 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:02.964925 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:02.964928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:02.964930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:02.965024 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:02.965357 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:02.965359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:02.965360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:02.965536 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:02.965609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:02.966201 (+ 592us) server_negotiation.cc:299] Negotiation successful
0809 03:34:02.966299 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:02.967494 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:02.955270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38859 (local address 127.15.222.65:54175)
0809 03:34:02.955684 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:02.955725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:02.956851 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:02.957306 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:02.957312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:02.957641 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:02.958153 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:02.958167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.958915 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.958918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:02.959801 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:02.959810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:02.960445 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:02.960450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:02.960606 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:02.961352 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:02.961371 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:02.962879 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:02.964561 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:02.964566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:02.964577 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:02.964824 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:02.965106 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:02.965108 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:02.965109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:02.965269 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:02.965630 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:02.965636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:02.965968 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 03:34:02.966160 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 03:34:02.966466 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260809 03:34:02.968672 10784 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38859
I20260809 03:34:02.968896 10784 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:02.969348 10784 heartbeater.cc:511] Master 127.15.222.126:38859 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
I20260809 03:34:02.971024 10507 ts_manager.cc:194] Registered new tserver with Master: 81b59ebf8e674fc698a4c27fd26b4d7b (127.15.222.65:44757)
I20260809 03:34:02.972244 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:54175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:34:02.986665 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:03.096175 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:03.096418 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:03.096458 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:03.100373 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:03.100436 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:03.100490 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:03.105553 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:03.106535 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:03.107527 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:03.116721 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:03.116717 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:03.116894 10791 server_base.cc:1034] running on GCE node
W20260809 03:34:03.116751 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:03.117350 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:03.117971 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:03.119124 10791 hybrid_clock.cc:648] HybridClock initialized: now 1786246443119106 us; error 28 us; skew 500 ppm
Aug 09 03:34:03 dist-test-slave-cvwc krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246443, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:03.121691 10791 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:34:03.122535 10791 webserver.cc:460] Webserver started at http://127.15.222.66:35993/ using document root <none> and password file <none>
I20260809 03:34:03.123062 10791 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:03.123121 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:03.123313 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:03.125296 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "3bcab4ab64ff476480a456927de370b2"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "0822a8a8868e55c4e8165f55e5dbba74"
server_key_iv: "1d4d25a5e080c7e799b0427c51172a49"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.125878 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "3bcab4ab64ff476480a456927de370b2"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "0822a8a8868e55c4e8165f55e5dbba74"
server_key_iv: "1d4d25a5e080c7e799b0427c51172a49"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.129297 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:34:03.131557 10813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.132661 10791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:03.132756 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "3bcab4ab64ff476480a456927de370b2"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "0822a8a8868e55c4e8165f55e5dbba74"
server_key_iv: "1d4d25a5e080c7e799b0427c51172a49"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.132846 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:03.150853 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:03.153596 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:03.153784 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:03.154349 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:03.155160 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:03.155206 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.155251 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:03.155280 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.163854 10791 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:35013
I20260809 03:34:03.163873 10985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:35013 every 8 connection(s)
I20260809 03:34:03.164737 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:34:03.172616 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 10791
I20260809 03:34:03.172715 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:03.172992 16249 external_mini_cluster.cc:1428] Setting key 22088282aca47feec23c757fcff1905e
Aug 09 03:34:03 dist-test-slave-cvwc krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246443, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:03.177870 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.166494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:46281 (local address 127.15.222.126:38859)
0809 03:34:03.166621 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.166625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.167426 (+ 801us) server_negotiation.cc:407] Connection header received
0809 03:34:03.168727 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.168731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.168791 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.168868 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:03.170118 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.170650 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.171790 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.172020 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.172111 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.174633 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:03.174636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:03.174638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:03.174661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:03.175959 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.176425 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.176428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.176429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.176506 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.176813 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.176816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.176818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.176995 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:03.177079 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.177561 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.177686 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:03.178660 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.166858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38859 (local address 127.15.222.66:46281)
0809 03:34:03.167282 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.167325 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.168514 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.169049 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.169056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.169404 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:03.169973 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.169994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.170769 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.170773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.171663 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.171673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.172160 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.172164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.172331 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.172918 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:03.172934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:03.174483 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:03.176067 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:03.176072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:03.176086 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:03.176326 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:03.176588 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:03.176591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:03.176593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:03.176728 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:03.177092 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:03.177099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:03.177333 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.177497 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.177837 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260809 03:34:03.179833 10986 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38859
I20260809 03:34:03.180050 10986 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:03.180521 10986 heartbeater.cc:511] Master 127.15.222.126:38859 requested a full tablet report, sending...
I20260809 03:34:03.181576 10507 ts_manager.cc:194] Registered new tserver with Master: 3bcab4ab64ff476480a456927de370b2 (127.15.222.66:35013)
I20260809 03:34:03.182353 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:46281
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:34:03.198489 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:03.309054 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:03.309289 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:03.309329 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:03.313326 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:03.313380 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:03.313449 10991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:03.318517 10991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:03.319409 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:03.320326 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:03.329604 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:03.329638 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:03.329659 11004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:03.329809 10991 server_base.cc:1034] running on GCE node
I20260809 03:34:03.330250 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:03.330824 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:03.332010 10991 hybrid_clock.cc:648] HybridClock initialized: now 1786246443331989 us; error 35 us; skew 500 ppm
Aug 09 03:34:03 dist-test-slave-cvwc krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246443, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:03.334574 10991 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:34:03.335423 10991 webserver.cc:460] Webserver started at http://127.15.222.67:40603/ using document root <none> and password file <none>
I20260809 03:34:03.335964 10991 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:03.336035 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:03.336228 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:03.338205 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "5904188f7b2b4f3b94131f18a8251282"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "bbce40e8bf4f8e6a4b0042fed237497b"
server_key_iv: "d29909927a3187463d552ae3dcba1796"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.338742 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "5904188f7b2b4f3b94131f18a8251282"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "bbce40e8bf4f8e6a4b0042fed237497b"
server_key_iv: "d29909927a3187463d552ae3dcba1796"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.342168 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:03.344369 11019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.345479 10991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:03.345582 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "5904188f7b2b4f3b94131f18a8251282"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "bbce40e8bf4f8e6a4b0042fed237497b"
server_key_iv: "d29909927a3187463d552ae3dcba1796"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.345669 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:03.358728 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:03.361557 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:03.361749 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:03.362291 10991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:03.363077 10991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:03.363124 10991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.363175 10991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:03.363202 10991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.372205 10991 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:45369
I20260809 03:34:03.372221 11187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:45369 every 8 connection(s)
I20260809 03:34:03.373085 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:03.374497 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 10991
I20260809 03:34:03.374634 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:03.375003 16249 external_mini_cluster.cc:1428] Setting key 91e46ac29565a440612a68d4f81d6351
Aug 09 03:34:03 dist-test-slave-cvwc krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246443, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:03.385505 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.374893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:47095 (local address 127.15.222.126:38859)
0809 03:34:03.375014 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.375018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.376016 (+ 998us) server_negotiation.cc:407] Connection header received
0809 03:34:03.377327 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.377329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.377376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.377434 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:03.378597 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.379111 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.380196 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.380396 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.380464 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.382843 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:03.382847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:03.382849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:03.382873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:03.383984 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.384390 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.384393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.384396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.384460 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.384697 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.384699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.384700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.384854 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:03.384930 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.385301 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.385389 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:34:03.386263 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.375370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38859 (local address 127.15.222.67:47095)
0809 03:34:03.375852 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.375907 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.377122 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.377586 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.377593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.377977 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:03.378468 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.378497 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.379209 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.379212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.380102 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.380111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.380545 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.380548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.380699 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.381230 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:03.381245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:03.382692 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:03.384076 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:03.384080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:03.384091 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:03.384319 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:03.384518 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:03.384520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:03.384522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:03.384637 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:03.384922 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:03.384927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:03.385131 (+ 204us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.385272 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.385507 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260809 03:34:03.387264 11189 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38859
I20260809 03:34:03.387486 11189 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:03.387984 11189 heartbeater.cc:511] Master 127.15.222.126:38859 requested a full tablet report, sending...
I20260809 03:34:03.388921 10507 ts_manager.cc:194] Registered new tserver with Master: 5904188f7b2b4f3b94131f18a8251282 (127.15.222.67:45369)
I20260809 03:34:03.389442 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:47095
I20260809 03:34:03.399930 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:03.407977 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.401702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.15.222.126:38859)
0809 03:34:03.401842 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.401846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.401989 (+ 143us) server_negotiation.cc:407] Connection header received
0809 03:34:03.402076 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.402078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.402114 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.402172 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:03.402734 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.403224 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.403972 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.404198 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.404263 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.405378 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:03.405381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:03.405384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:03.405407 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:03.406664 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.406972 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.406974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.406975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.407045 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:03.407295 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:03.407297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:03.407298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:03.407493 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:03.407589 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.407740 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.407825 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 03:34:03.410501 10507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:03.413347 10507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:03.426493 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.420992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:45369 (local address 127.0.0.1:46672)
0809 03:34:03.421328 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.421380 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.421768 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.422819 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.422825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.422863 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:03.423049 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.423057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424124 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.424127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.425158 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.425169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.425987 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.425991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.426097 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.426133 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.426232 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.426350 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":64,"spinlock_wait_cycles":10112,"thread_start_us":97,"threads_started":1}
I20260809 03:34:03.426497 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.421999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.15.222.67:45369)
0809 03:34:03.422368 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.422380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.422402 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:34:03.422475 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.422479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.422666 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.422950 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:03.423153 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424006 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.425286 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.425878 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.425975 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.426118 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.426271 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.426362 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":159,"threads_started":1}
I20260809 03:34:03.427219 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.421083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:35013 (local address 127.0.0.1:42726)
0809 03:34:03.421972 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.421982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.422055 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.422821 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.422826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.422861 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:03.423056 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.423064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424645 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.424648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.425589 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.425599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.426852 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.426855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.426946 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.426966 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.427024 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.427090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":26,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260809 03:34:03.427251 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.421557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.15.222.66:35013)
0809 03:34:03.421963 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.421975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.422052 (+ 77us) server_negotiation.cc:407] Connection header received
0809 03:34:03.422152 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.422158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.422639 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.423202 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:03.423352 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424475 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.425703 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.426546 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.426859 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.427009 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.427056 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.427133 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":158,"threads_started":1}
I20260809 03:34:03.428052 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.421688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44757 (local address 127.0.0.1:51444)
0809 03:34:03.422664 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.422676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.422756 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.423341 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.423344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.423363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:03.423447 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.423453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424753 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.424756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.426691 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.426701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.427695 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.427699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.427781 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.427800 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.427874 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.427937 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":106,"threads_started":1}
I20260809 03:34:03.428118 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.422536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.15.222.65:44757)
0809 03:34:03.422931 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.422943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.422964 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:34:03.423036 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.423041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.423211 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.423351 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:03.423563 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.424628 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.426810 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.427587 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.427682 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.427811 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.427909 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.427998 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":109,"threads_started":1}
I20260809 03:34:03.429838 11088 tablet_service.cc:1467] Processing CreateTablet for tablet aadea45268b24dfeb80b42bcdcbe49af (DEFAULT_TABLE table=test-table [id=2ef835835dce4a06a174b0f035833adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:03.430696 10688 tablet_service.cc:1467] Processing CreateTablet for tablet aadea45268b24dfeb80b42bcdcbe49af (DEFAULT_TABLE table=test-table [id=2ef835835dce4a06a174b0f035833adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:03.430986 11088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadea45268b24dfeb80b42bcdcbe49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:03.431083 10881 tablet_service.cc:1467] Processing CreateTablet for tablet aadea45268b24dfeb80b42bcdcbe49af (DEFAULT_TABLE table=test-table [id=2ef835835dce4a06a174b0f035833adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:03.431489 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadea45268b24dfeb80b42bcdcbe49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:03.432039 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadea45268b24dfeb80b42bcdcbe49af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:03.437187 11210 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Bootstrap starting.
I20260809 03:34:03.439165 11214 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Bootstrap starting.
I20260809 03:34:03.439368 11213 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Bootstrap starting.
I20260809 03:34:03.439522 11210 tablet_bootstrap.cc:654] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:03.440843 11210 log.cc:824] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:03.441108 11213 tablet_bootstrap.cc:654] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:03.441264 11214 tablet_bootstrap.cc:654] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:03.442224 11213 log.cc:824] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:03.442399 11214 log.cc:824] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:03.443045 11210 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: No bootstrap required, opened a new log
I20260809 03:34:03.443235 11210 ts_tablet_manager.cc:1397] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:34:03.444067 11213 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: No bootstrap required, opened a new log
I20260809 03:34:03.444165 11214 tablet_bootstrap.cc:492] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: No bootstrap required, opened a new log
I20260809 03:34:03.444245 11213 ts_tablet_manager.cc:1397] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:34:03.444336 11214 ts_tablet_manager.cc:1397] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 03:34:03.447078 11210 raft_consensus.cc:348] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.447149 11214 raft_consensus.cc:348] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.447345 11214 raft_consensus.cc:374] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:03.447346 11210 raft_consensus.cc:374] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:03.447413 11210 raft_consensus.cc:729] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b59ebf8e674fc698a4c27fd26b4d7b, State: Initialized, Role: FOLLOWER
I20260809 03:34:03.447413 11214 raft_consensus.cc:729] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcab4ab64ff476480a456927de370b2, State: Initialized, Role: FOLLOWER
I20260809 03:34:03.447806 11214 consensus_queue.cc:260] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.447856 11210 consensus_queue.cc:260] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.447788 11213 raft_consensus.cc:348] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.448006 11213 raft_consensus.cc:374] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:03.448067 11213 raft_consensus.cc:729] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5904188f7b2b4f3b94131f18a8251282, State: Initialized, Role: FOLLOWER
I20260809 03:34:03.448443 11213 consensus_queue.cc:260] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.448709 11214 ts_tablet_manager.cc:1428] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260809 03:34:03.448783 11210 ts_tablet_manager.cc:1428] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:34:03.449110 10986 heartbeater.cc:503] Master 127.15.222.126:38859 was elected leader, sending a full tablet report...
I20260809 03:34:03.449110 10784 heartbeater.cc:503] Master 127.15.222.126:38859 was elected leader, sending a full tablet report...
I20260809 03:34:03.449270 11213 ts_tablet_manager.cc:1428] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:34:03.449872 11189 heartbeater.cc:503] Master 127.15.222.126:38859 was elected leader, sending a full tablet report...
W20260809 03:34:03.454249 10785 tablet.cc:2369] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:03.587334 11222 raft_consensus.cc:482] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:03.587584 11222 raft_consensus.cc:504] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.588762 11222 leader_election.cc:288] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5904188f7b2b4f3b94131f18a8251282 (127.15.222.67:45369), 81b59ebf8e674fc698a4c27fd26b4d7b (127.15.222.65:44757)
I20260809 03:34:03.592904 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.589037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:45369 (local address 127.15.222.66:40737)
0809 03:34:03.589155 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.589171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.589269 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.589516 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.589519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.589542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:03.589666 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.589673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.590658 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.590661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.591800 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.591810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.592541 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.592544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.592622 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.592643 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.592699 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.592759 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 03:34:03.592933 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.589136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40737 (local address 127.15.222.67:45369)
0809 03:34:03.589278 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.589283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.589300 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:34:03.589354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.589357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.589408 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.589485 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:03.589836 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.590553 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.591922 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.592435 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.592546 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.592600 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.592734 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.592801 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 03:34:03.593168 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.589037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44757 (local address 127.15.222.66:49759)
0809 03:34:03.589705 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.589739 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:03.589832 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:03.590050 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:03.590052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:03.590064 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:03.590160 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.590164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.590965 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.590969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.592031 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:03.592039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.592795 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.592799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:03.592887 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.592909 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:34:03.592979 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:34:03.593043 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":295,"thread_start_us":153,"threads_started":1}
I20260809 03:34:03.593588 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.589233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:49759 (local address 127.15.222.65:44757)
0809 03:34:03.589354 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.589357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.589824 (+ 467us) server_negotiation.cc:407] Connection header received
0809 03:34:03.589912 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.589915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.589966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.590043 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:03.590249 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.590862 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.592137 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.592680 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.592789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.592833 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.593407 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.593479 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:03.593539 11117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadea45268b24dfeb80b42bcdcbe49af" candidate_uuid: "3bcab4ab64ff476480a456927de370b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5904188f7b2b4f3b94131f18a8251282" is_pre_election: true
I20260809 03:34:03.593858 11117 raft_consensus.cc:2454] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcab4ab64ff476480a456927de370b2 in term 0.
I20260809 03:34:03.594180 10719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadea45268b24dfeb80b42bcdcbe49af" candidate_uuid: "3bcab4ab64ff476480a456927de370b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" is_pre_election: true
I20260809 03:34:03.594327 10814 leader_election.cc:302] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcab4ab64ff476480a456927de370b2, 5904188f7b2b4f3b94131f18a8251282; no voters:
I20260809 03:34:03.594478 10719 raft_consensus.cc:2454] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcab4ab64ff476480a456927de370b2 in term 0.
I20260809 03:34:03.594607 11222 raft_consensus.cc:2781] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:03.594703 11222 raft_consensus.cc:482] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:03.594755 11222 raft_consensus.cc:3037] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:03.595840 11222 raft_consensus.cc:504] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.596163 11222 leader_election.cc:288] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5904188f7b2b4f3b94131f18a8251282 (127.15.222.67:45369), 81b59ebf8e674fc698a4c27fd26b4d7b (127.15.222.65:44757)
I20260809 03:34:03.596513 11117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadea45268b24dfeb80b42bcdcbe49af" candidate_uuid: "3bcab4ab64ff476480a456927de370b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5904188f7b2b4f3b94131f18a8251282"
I20260809 03:34:03.596513 10719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadea45268b24dfeb80b42bcdcbe49af" candidate_uuid: "3bcab4ab64ff476480a456927de370b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b"
I20260809 03:34:03.596640 10719 raft_consensus.cc:3037] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:03.596640 11117 raft_consensus.cc:3037] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:03.597936 10719 raft_consensus.cc:2454] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcab4ab64ff476480a456927de370b2 in term 1.
I20260809 03:34:03.597939 11117 raft_consensus.cc:2454] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bcab4ab64ff476480a456927de370b2 in term 1.
I20260809 03:34:03.598292 10818 leader_election.cc:302] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bcab4ab64ff476480a456927de370b2, 5904188f7b2b4f3b94131f18a8251282, 81b59ebf8e674fc698a4c27fd26b4d7b; no voters:
I20260809 03:34:03.598469 11222 raft_consensus.cc:2781] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:03.598703 11222 raft_consensus.cc:686] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3bcab4ab64ff476480a456927de370b2, State: Running, Role: LEADER
I20260809 03:34:03.599045 11222 consensus_queue.cc:237] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } }
I20260809 03:34:03.602155 10507 catalog_manager.cc:5495] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bcab4ab64ff476480a456927de370b2 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "3bcab4ab64ff476480a456927de370b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bcab4ab64ff476480a456927de370b2" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 } health_report { overall_health: UNKNOWN } } }
I20260809 03:34:03.613698 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.610731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.15.222.66:35013)
0809 03:34:03.610830 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.610833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.610859 (+ 26us) server_negotiation.cc:407] Connection header received
0809 03:34:03.611013 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.611015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.611065 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.611149 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:03.611316 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.611799 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.612546 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.612739 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.612803 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.613019 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:03.613414 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:03.613496 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.613535 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.613580 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:34:03.619105 11117 raft_consensus.cc:1261] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 1 FOLLOWER]: Refusing update from remote peer 3bcab4ab64ff476480a456927de370b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:03.619105 10719 raft_consensus.cc:1261] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 1 FOLLOWER]: Refusing update from remote peer 3bcab4ab64ff476480a456927de370b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:03.619738 11226 consensus_queue.cc:1035] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b59ebf8e674fc698a4c27fd26b4d7b" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:03.619910 11222 consensus_queue.cc:1035] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5904188f7b2b4f3b94131f18a8251282" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 03:34:03.623974 11191 tablet.cc:2369] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:03.635650 10505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:03.635875 10505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:03.638242 10505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 810fe1cbbbba4194b2e418fc8b57f2c1: Sending DeleteTablet for 3 replicas of tablet aadea45268b24dfeb80b42bcdcbe49af
I20260809 03:34:03.639052 11088 tablet_service.cc:1514] Processing DeleteTablet for tablet aadea45268b24dfeb80b42bcdcbe49af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:03 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:46672
I20260809 03:34:03.639072 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet aadea45268b24dfeb80b42bcdcbe49af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:03 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:51444
I20260809 03:34:03.639534 11245 tablet_replica.cc:330] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: stopping tablet replica
I20260809 03:34:03.639534 11247 tablet_replica.cc:330] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b: stopping tablet replica
I20260809 03:34:03.639767 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet aadea45268b24dfeb80b42bcdcbe49af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:03 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:42726
I20260809 03:34:03.639835 11247 raft_consensus.cc:2229] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:03.639835 11245 raft_consensus.cc:2229] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:03.640153 11247 raft_consensus.cc:2258] T aadea45268b24dfeb80b42bcdcbe49af P 81b59ebf8e674fc698a4c27fd26b4d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:03.640163 11245 raft_consensus.cc:2258] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:03.640532 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 10597
I20260809 03:34:03.641918 11245 ts_tablet_manager.cc:1905] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:03.645320 11245 ts_tablet_manager.cc:1918] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:03.645520 11245 log.cc:1196] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/aadea45268b24dfeb80b42bcdcbe49af
I20260809 03:34:03.646145 11245 ts_tablet_manager.cc:1939] T aadea45268b24dfeb80b42bcdcbe49af P 5904188f7b2b4f3b94131f18a8251282: Deleting consensus metadata
I20260809 03:34:03.653240 10484 catalog_manager.cc:4841] TS 5904188f7b2b4f3b94131f18a8251282 (127.15.222.67:45369): tablet aadea45268b24dfeb80b42bcdcbe49af (table test-table [id=2ef835835dce4a06a174b0f035833adb]) successfully deleted
I20260809 03:34:03.657979 11248 tablet_replica.cc:330] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: stopping tablet replica
I20260809 03:34:03.658339 11248 raft_consensus.cc:2229] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:34:03.658728 11248 raft_consensus.cc:2258] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:03.660360 11248 ts_tablet_manager.cc:1905] T aadea45268b24dfeb80b42bcdcbe49af P 3bcab4ab64ff476480a456927de370b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:34:03.661034 10486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:44757 (error 108)
I20260809 03:34:03.661120 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 10791
I20260809 03:34:03.667603 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.667282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44757 (local address 127.0.0.1:51448)
0809 03:34:03.667457 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.667521 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260809 03:34:03.668013 10486 catalog_manager.cc:4584] TS 81b59ebf8e674fc698a4c27fd26b4d7b (127.15.222.65:44757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aadea45268b24dfeb80b42bcdcbe49af: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44757: connect: Connection refused (error 111)
I20260809 03:34:03.676247 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 10991
I20260809 03:34:03.703423 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.676388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:35013 (local address 127.0.0.1:42740)
0809 03:34:03.703192 (+ 26804us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:03.703308 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.66:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26667}
I20260809 03:34:03.709265 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 10457
2026-08-09T03:34:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2465 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T03:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:03Z Disabled control of system clock
I20260809 03:34:03.749254 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43301
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:03.886895 11256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:03.887148 11256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:03.887180 11256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:03.891010 11256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:03.891062 11256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:03.891075 11256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:03.891089 11256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:03.891103 11256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:03.895702 11256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:43301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:43301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:03.896704 11256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:03.898026 11256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:03.906378 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:03.906378 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:03.906538 11256 server_base.cc:1034] running on GCE node
W20260809 03:34:03.906553 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:03.907236 11256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:03.908282 11256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:03.909438 11256 hybrid_clock.cc:648] HybridClock initialized: now 1786246443909428 us; error 27 us; skew 500 ppm
I20260809 03:34:03.911152 11256 webserver.cc:460] Webserver started at http://127.15.222.126:46429/ using document root <none> and password file <none>
I20260809 03:34:03.911705 11256 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:03.911770 11256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:03.911960 11256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:03.913982 11256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3b425cd5f9f6403daaf452116a038691"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "5f455de2539a0def6608c0d36dd6a36c"
server_key_iv: "e1f28d954e8db7ad5090722c626cdfe4"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.914520 11256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3b425cd5f9f6403daaf452116a038691"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "5f455de2539a0def6608c0d36dd6a36c"
server_key_iv: "e1f28d954e8db7ad5090722c626cdfe4"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.918248 11256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 03:34:03.920972 11282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:03.922132 11256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:03.922235 11256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3b425cd5f9f6403daaf452116a038691"
format_stamp: "Formatted at 2026-08-09 03:34:03 on dist-test-slave-cvwc"
server_key: "5f455de2539a0def6608c0d36dd6a36c"
server_key_iv: "e1f28d954e8db7ad5090722c626cdfe4"
server_key_version: "encryptionkey@0"
I20260809 03:34:03.922335 11256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:03.945770 11256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:03.946503 11256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:03.946689 11256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:03.953408 11256 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:43301
I20260809 03:34:03.953424 11359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:43301 every 8 connection(s)
I20260809 03:34:03.954376 11256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:03.955813 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 11256
I20260809 03:34:03.955938 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:03.956207 16249 external_mini_cluster.cc:1428] Setting key 756f77c879b027c54c22eaf947fc8946
I20260809 03:34:03.957803 11360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:03.964766 11360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: Bootstrap starting.
I20260809 03:34:03.966562 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:03.957586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.15.222.126:43301)
0809 03:34:03.958271 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:03.958291 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:03.958333 (+ 42us) server_negotiation.cc:407] Connection header received
0809 03:34:03.959181 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:03.959203 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:03.959694 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:03.960064 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:03.960532 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.962001 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.962889 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:03.963585 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:03.963754 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:03.964488 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:03.964495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:03.964511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:03.965002 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:03.965072 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:03.965099 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:03.965196 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:03.965316 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 03:34:03.965590 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":159,"threads_started":1}
I20260809 03:34:03.967419 11360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:03.968305 11360 log.cc:824] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:03.970311 11360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: No bootstrap required, opened a new log
I20260809 03:34:03.973821 11360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } }
I20260809 03:34:03.974067 11360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:03.974126 11360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b425cd5f9f6403daaf452116a038691, State: Initialized, Role: FOLLOWER
I20260809 03:34:03.974555 11360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } }
I20260809 03:34:03.974691 11360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:03.974756 11360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:03.974854 11360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:03.975797 11360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } }
I20260809 03:34:03.976137 11360 leader_election.cc:302] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b425cd5f9f6403daaf452116a038691; no voters:
I20260809 03:34:03.976454 11360 leader_election.cc:288] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:03.976562 11365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:03.976763 11365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [term 1 LEADER]: Becoming Leader. State: Replica: 3b425cd5f9f6403daaf452116a038691, State: Running, Role: LEADER
I20260809 03:34:03.977092 11365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } }
I20260809 03:34:03.977741 11360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:03.978850 11368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b425cd5f9f6403daaf452116a038691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } } }
I20260809 03:34:03.978850 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b425cd5f9f6403daaf452116a038691. Latest consensus state: current_term: 1 leader_uuid: "3b425cd5f9f6403daaf452116a038691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b425cd5f9f6403daaf452116a038691" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 43301 } } }
I20260809 03:34:03.979017 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:03.979223 11368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:03.979373 11378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:03.982664 11378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:03.988531 11378 catalog_manager.cc:1347] Generated new cluster ID: 246a669dab024eb4ba5b9b6c9898fc28
I20260809 03:34:03.988592 11378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:03.996383 11378 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:03.997593 11378 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:04.005800 11378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: Generated new TSK 0
I20260809 03:34:04.006608 11378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:04.015223 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--builtin_ntp_servers=127.15.222.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:04.127043 11391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:04.127282 11391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:04.127332 11391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:04.131256 11391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:04.131300 11391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:04.131373 11391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:04.135938 11391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:04.136879 11391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:04.138258 11391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:04.147352 11406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.147390 11391 server_base.cc:1034] running on GCE node
W20260809 03:34:04.147433 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:04.147367 11397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.147907 11391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:04.148464 11391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:04.149623 11391 hybrid_clock.cc:648] HybridClock initialized: now 1786246444149605 us; error 27 us; skew 500 ppm
I20260809 03:34:04.151297 11391 webserver.cc:460] Webserver started at http://127.15.222.65:41567/ using document root <none> and password file <none>
I20260809 03:34:04.151819 11391 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:04.151893 11391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:04.152089 11391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:04.154034 11391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "0422aead5624494d839898bb6d2b4db7"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "c9ea2bdd28ac3c3a8cebaafe955f4d39"
server_key_iv: "2e0206cec220d85fd89d7648d8daac8a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.154543 11391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "0422aead5624494d839898bb6d2b4db7"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "c9ea2bdd28ac3c3a8cebaafe955f4d39"
server_key_iv: "2e0206cec220d85fd89d7648d8daac8a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.157872 11391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:04.160076 11413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.161170 11391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:04.161271 11391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "0422aead5624494d839898bb6d2b4db7"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "c9ea2bdd28ac3c3a8cebaafe955f4d39"
server_key_iv: "2e0206cec220d85fd89d7648d8daac8a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.161358 11391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:04.183557 11391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:04.184248 11391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:04.184429 11391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:04.185009 11391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:04.185964 11391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:04.186012 11391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.186065 11391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:04.186093 11391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.194183 11391 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:41925
I20260809 03:34:04.194216 11575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:41925 every 8 connection(s)
I20260809 03:34:04.195057 11391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:04.202231 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 11391
I20260809 03:34:04.202343 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:34:04.202612 16249 external_mini_cluster.cc:1428] Setting key e3c001f702861610a6c180d4bf756713
I20260809 03:34:04.204079 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.196776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:41755 (local address 127.15.222.126:43301)
0809 03:34:04.196921 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.196925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.197945 (+ 1020us) server_negotiation.cc:407] Connection header received
0809 03:34:04.199387 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.199398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.199461 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.199534 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:04.200431 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.200967 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.202196 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.202474 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.202569 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.203616 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:04.203620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:04.203622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:04.203635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:04.203655 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:04.203663 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:04.203720 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.203864 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.203941 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 03:34:04.204514 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--builtin_ntp_servers=127.15.222.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:04.204926 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.197149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43301 (local address 127.15.222.65:41755)
0809 03:34:04.197792 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.197833 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.199167 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.199717 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.199724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.199808 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:04.200281 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.200293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.201090 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.201093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.202076 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.202086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.202622 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.202627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.202811 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.203368 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:04.203370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:04.203410 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:04.203473 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:04.203730 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:04.203738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:04.203745 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.203851 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.204138 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260809 03:34:04.206070 11577 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43301
I20260809 03:34:04.206321 11577 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:04.206801 11577 heartbeater.cc:511] Master 127.15.222.126:43301 requested a full tablet report, sending...
I20260809 03:34:04.208344 11304 ts_manager.cc:194] Registered new tserver with Master: 0422aead5624494d839898bb6d2b4db7 (127.15.222.65:41925)
I20260809 03:34:04.209946 11304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.65:41755
W20260809 03:34:04.313853 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:04.314077 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:04.314110 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:04.317888 11583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:04.317924 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:04.317972 11583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:04.322396 11583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:04.323259 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:04.324496 11583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:04.333588 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.333730 11583 server_base.cc:1034] running on GCE node
W20260809 03:34:04.333611 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:04.333643 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.334205 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:04.334776 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:04.335924 11583 hybrid_clock.cc:648] HybridClock initialized: now 1786246444335906 us; error 27 us; skew 500 ppm
I20260809 03:34:04.337579 11583 webserver.cc:460] Webserver started at http://127.15.222.66:43431/ using document root <none> and password file <none>
I20260809 03:34:04.338126 11583 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:04.338186 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:04.338403 11583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:04.340344 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "72263a1203b74069a145188b22ad64db"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "dfcf31e0ab0833a0ae31afbfb9ddf1a8"
server_key_iv: "aa1da71c9908fc84e43f64111ac4763a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.340886 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "72263a1203b74069a145188b22ad64db"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "dfcf31e0ab0833a0ae31afbfb9ddf1a8"
server_key_iv: "aa1da71c9908fc84e43f64111ac4763a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.344383 11583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 03:34:04.346568 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.347664 11583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:04.347752 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "72263a1203b74069a145188b22ad64db"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "dfcf31e0ab0833a0ae31afbfb9ddf1a8"
server_key_iv: "aa1da71c9908fc84e43f64111ac4763a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.347831 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:04.376493 11583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:04.377178 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:04.377332 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:04.377986 11583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:04.379034 11583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:04.379083 11583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.379124 11583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:04.379140 11583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.387138 11583 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:35147
I20260809 03:34:04.387166 11766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:35147 every 8 connection(s)
I20260809 03:34:04.388006 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:34:04.391340 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 11583
I20260809 03:34:04.391451 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:04.391718 16249 external_mini_cluster.cc:1428] Setting key f5e51bca8122198a841b859593f7db82
I20260809 03:34:04.393757 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--builtin_ntp_servers=127.15.222.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:04.397115 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.389708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:37247 (local address 127.15.222.126:43301)
0809 03:34:04.389848 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.389852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.390917 (+ 1065us) server_negotiation.cc:407] Connection header received
0809 03:34:04.392363 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.392366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.392420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.392501 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:04.393533 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.394242 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.395341 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.395553 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.395640 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.396665 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:04.396668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:04.396669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:04.396680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:04.396700 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:04.396707 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:04.396765 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.396912 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.396978 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:34:04.397928 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.390141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43301 (local address 127.15.222.66:37247)
0809 03:34:04.390777 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.390817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.392166 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.392692 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.392701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.392798 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:04.393396 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.393410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.394356 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.394359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.395217 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.395226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.395685 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.395689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.395842 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.396419 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:04.396420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:04.396458 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:04.396519 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:04.396774 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:04.396782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:04.396789 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.396893 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.397164 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260809 03:34:04.398929 11767 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43301
I20260809 03:34:04.399168 11767 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:04.399757 11767 heartbeater.cc:511] Master 127.15.222.126:43301 requested a full tablet report, sending...
I20260809 03:34:04.400805 11304 ts_manager.cc:194] Registered new tserver with Master: 72263a1203b74069a145188b22ad64db (127.15.222.66:35147)
I20260809 03:34:04.401338 11304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.66:37247
W20260809 03:34:04.504602 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:04.504837 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:04.504873 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:04.509054 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:04.509105 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:04.509162 11773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:04.513816 11773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:43301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:04.514787 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:04.516094 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:04.525360 11786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:04.525364 11785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.525503 11773 server_base.cc:1034] running on GCE node
W20260809 03:34:04.525733 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:04.526100 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:04.526718 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:04.527878 11773 hybrid_clock.cc:648] HybridClock initialized: now 1786246444527864 us; error 31 us; skew 500 ppm
I20260809 03:34:04.529546 11773 webserver.cc:460] Webserver started at http://127.15.222.67:39999/ using document root <none> and password file <none>
I20260809 03:34:04.530115 11773 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:04.530182 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:04.530375 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:04.532328 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "cdd61df92fc24247b56af0b50c5c477e"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "94b6d42eb2369607a939a6f71cc4469f"
server_key_iv: "a14a07e1bf8aebfd385dfd337e06ca6a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.532858 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "cdd61df92fc24247b56af0b50c5c477e"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "94b6d42eb2369607a939a6f71cc4469f"
server_key_iv: "a14a07e1bf8aebfd385dfd337e06ca6a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.536337 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:04.538561 11799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.539685 11773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:04.539783 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "cdd61df92fc24247b56af0b50c5c477e"
format_stamp: "Formatted at 2026-08-09 03:34:04 on dist-test-slave-cvwc"
server_key: "94b6d42eb2369607a939a6f71cc4469f"
server_key_iv: "a14a07e1bf8aebfd385dfd337e06ca6a"
server_key_version: "encryptionkey@0"
I20260809 03:34:04.539870 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:04.550938 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:04.551571 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:04.551724 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:04.552246 11773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:04.553125 11773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:04.553171 11773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.553222 11773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:04.553247 11773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:04.561419 11773 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:41347
I20260809 03:34:04.561448 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:41347 every 8 connection(s)
I20260809 03:34:04.562320 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:04.570238 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 11773
I20260809 03:34:04.570330 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:04.570641 16249 external_mini_cluster.cc:1428] Setting key be9cfe04981cbc2d83138cdd36ee6cb5
I20260809 03:34:04.571012 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:53647 (local address 127.15.222.126:43301)
0809 03:34:04.564172 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.564176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.565236 (+ 1060us) server_negotiation.cc:407] Connection header received
0809 03:34:04.566671 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.566673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.566719 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.566779 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:04.567614 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.568128 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.569151 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.569374 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.569460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.570516 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:04.570520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:04.570523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:04.570537 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:04.570559 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:04.570569 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:04.570641 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.570801 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.570861 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 03:34:04.571866 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.564462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:43301 (local address 127.15.222.67:53647)
0809 03:34:04.565097 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.565137 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.566471 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.566932 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.566939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.567018 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:04.567509 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.567523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.568214 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.568216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.569066 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.569075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.569510 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.569513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.569650 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.570243 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:04.570245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:04.570282 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:04.570363 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:04.570640 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:04.570645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:04.570649 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.570755 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.570981 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260809 03:34:04.572901 11963 heartbeater.cc:347] Connected to a master server at 127.15.222.126:43301
I20260809 03:34:04.573099 11963 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:04.573556 11963 heartbeater.cc:511] Master 127.15.222.126:43301 requested a full tablet report, sending...
I20260809 03:34:04.574501 11304 ts_manager.cc:194] Registered new tserver with Master: cdd61df92fc24247b56af0b50c5c477e (127.15.222.67:41347)
I20260809 03:34:04.575031 11304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.67:53647
I20260809 03:34:04.584249 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:04.589677 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.586121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.15.222.126:43301)
0809 03:34:04.586239 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.586244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.586395 (+ 151us) server_negotiation.cc:407] Connection header received
0809 03:34:04.586535 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.586537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.586576 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.586634 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:04.586918 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.587408 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.588210 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.588420 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.588505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.589287 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:04.589290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:04.589292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:04.589303 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:04.589321 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:04.589328 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:04.589385 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.589488 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.589541 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:34:04.592235 11304 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:04.595293 11304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:04.609517 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.603649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.15.222.67:41347)
0809 03:34:04.604767 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.604777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.604797 (+ 20us) server_negotiation.cc:407] Connection header received
0809 03:34:04.604859 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.604864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.605003 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.605290 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:04.605508 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606783 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608157 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.608848 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608965 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609151 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.609252 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.609349 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":120,"threads_started":1}
I20260809 03:34:04.609509 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.603504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:41347 (local address 127.0.0.1:37270)
0809 03:34:04.603850 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.603897 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.604017 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.605149 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.605156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.605214 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:04.605398 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.605405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606881 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.606892 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.608042 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.608052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.608977 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.609083 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609121 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.609227 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.609342 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":88,"spinlock_wait_cycles":10240,"thread_start_us":90,"threads_started":1}
I20260809 03:34:04.609681 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.603733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:35147 (local address 127.0.0.1:39062)
0809 03:34:04.604442 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.604456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.604545 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.605149 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.605156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.605212 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:04.605379 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.605386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606920 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.606923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.607929 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.607940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.609230 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.609233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.609317 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609340 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.609419 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.609483 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":10,"spinlock_wait_cycles":54912,"thread_start_us":89,"threads_started":1}
I20260809 03:34:04.610178 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.604285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.15.222.66:35147)
0809 03:34:04.604638 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.604649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.604673 (+ 24us) server_negotiation.cc:407] Connection header received
0809 03:34:04.604749 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.604754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.604969 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.605517 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:04.605654 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606771 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608058 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.608846 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608965 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609151 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.609935 (+ 784us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.610034 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":131,"threads_started":1}
I20260809 03:34:04.610250 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.604400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.15.222.65:41925)
0809 03:34:04.604897 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.604910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.605116 (+ 206us) server_negotiation.cc:407] Connection header received
0809 03:34:04.605376 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.605381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.605571 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.605694 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:04.605901 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606623 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.608418 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.608952 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.609050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609243 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.610031 (+ 788us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.610126 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":103,"threads_started":1}
I20260809 03:34:04.610545 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.604226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:41925 (local address 127.0.0.1:34162)
0809 03:34:04.605025 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.605038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.605199 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.605682 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.605685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.605703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:04.605808 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.605814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.606725 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.606729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.608315 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.608322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.609302 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.609305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.609380 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.609401 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.609463 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.609522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":36,"spinlock_wait_cycles":4480,"thread_start_us":91,"threads_started":1}
I20260809 03:34:04.612576 11863 tablet_service.cc:1467] Processing CreateTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 (DEFAULT_TABLE table=test-table [id=f3d94c7bc6d74e508da085f576b219fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:04.612895 11487 tablet_service.cc:1467] Processing CreateTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 (DEFAULT_TABLE table=test-table [id=f3d94c7bc6d74e508da085f576b219fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:04.612861 11674 tablet_service.cc:1467] Processing CreateTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 (DEFAULT_TABLE table=test-table [id=f3d94c7bc6d74e508da085f576b219fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:04.613473 11863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee941cd780d4796b24ed2fda2c45fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:04.613900 11487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee941cd780d4796b24ed2fda2c45fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:04.613895 11674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee941cd780d4796b24ed2fda2c45fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:04.619982 11991 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Bootstrap starting.
I20260809 03:34:04.620601 11992 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Bootstrap starting.
I20260809 03:34:04.621490 11993 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Bootstrap starting.
I20260809 03:34:04.622272 11991 tablet_bootstrap.cc:654] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:04.622800 11992 tablet_bootstrap.cc:654] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:04.623171 11993 tablet_bootstrap.cc:654] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:04.623150 11991 log.cc:824] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:04.623570 11992 log.cc:824] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:04.623867 11993 log.cc:824] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:04.624912 11991 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: No bootstrap required, opened a new log
I20260809 03:34:04.625113 11991 ts_tablet_manager.cc:1397] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:34:04.625553 11993 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: No bootstrap required, opened a new log
I20260809 03:34:04.625754 11993 ts_tablet_manager.cc:1397] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260809 03:34:04.628340 11991 raft_consensus.cc:348] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.628556 11991 raft_consensus.cc:374] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:04.628589 11991 raft_consensus.cc:729] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd61df92fc24247b56af0b50c5c477e, State: Initialized, Role: FOLLOWER
I20260809 03:34:04.628798 11993 raft_consensus.cc:348] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.628995 11993 raft_consensus.cc:374] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:04.629050 11993 raft_consensus.cc:729] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0422aead5624494d839898bb6d2b4db7, State: Initialized, Role: FOLLOWER
I20260809 03:34:04.629005 11991 consensus_queue.cc:260] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.629467 11993 consensus_queue.cc:260] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.629662 11963 heartbeater.cc:503] Master 127.15.222.126:43301 was elected leader, sending a full tablet report...
I20260809 03:34:04.629906 11991 ts_tablet_manager.cc:1428] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 03:34:04.630218 11577 heartbeater.cc:503] Master 127.15.222.126:43301 was elected leader, sending a full tablet report...
I20260809 03:34:04.630414 11993 ts_tablet_manager.cc:1428] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260809 03:34:04.632690 11992 tablet_bootstrap.cc:492] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: No bootstrap required, opened a new log
I20260809 03:34:04.632885 11992 ts_tablet_manager.cc:1397] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260809 03:34:04.636242 11992 raft_consensus.cc:348] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.636457 11992 raft_consensus.cc:374] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:04.636513 11992 raft_consensus.cc:729] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263a1203b74069a145188b22ad64db, State: Initialized, Role: FOLLOWER
I20260809 03:34:04.636880 11992 consensus_queue.cc:260] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.637465 11767 heartbeater.cc:503] Master 127.15.222.126:43301 was elected leader, sending a full tablet report...
I20260809 03:34:04.637656 11992 ts_tablet_manager.cc:1428] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260809 03:34:04.638803 11771 tablet.cc:2369] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:34:04.696144 11579 tablet.cc:2369] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:04.721964 12001 raft_consensus.cc:482] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:04.722203 12001 raft_consensus.cc:504] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.723455 12001 leader_election.cc:288] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd61df92fc24247b56af0b50c5c477e (127.15.222.67:41347), 72263a1203b74069a145188b22ad64db (127.15.222.66:35147)
I20260809 03:34:04.728384 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.723740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:38841 (local address 127.15.222.67:41347)
0809 03:34:04.723858 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.723863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.724026 (+ 163us) server_negotiation.cc:407] Connection header received
0809 03:34:04.724174 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.724177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.724229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.724308 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:04.724655 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.725432 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727123 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.727735 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727854 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.727910 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.728196 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.728258 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:04.728396 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.723787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:57587 (local address 127.15.222.66:35147)
0809 03:34:04.723903 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.723907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.724086 (+ 179us) server_negotiation.cc:407] Connection header received
0809 03:34:04.724183 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.724185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.724228 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.724308 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:04.724657 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.725432 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727123 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.727735 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727854 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.727907 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.728187 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.728255 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:34:04.728442 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.723645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:41347 (local address 127.15.222.65:38841)
0809 03:34:04.723933 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.723948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.724060 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.724324 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.724327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.724367 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:04.724547 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.724554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.725534 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.725537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.727007 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.727017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.727881 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.728044 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.728085 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.728166 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.728273 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":210,"spinlock_wait_cycles":32896,"thread_start_us":70,"threads_started":1}
I20260809 03:34:04.728442 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.723677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:35147 (local address 127.15.222.65:57587)
0809 03:34:04.724018 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.724030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:04.724105 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:04.724325 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:04.724327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:04.724369 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:04.724547 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.724554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.725534 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.725537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.727006 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:04.727017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.727883 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.727886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:04.728040 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.728089 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:34:04.728170 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 03:34:04.728262 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":113,"spinlock_wait_cycles":2944,"thread_start_us":87,"threads_started":1}
I20260809 03:34:04.729074 11894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee941cd780d4796b24ed2fda2c45fc9" candidate_uuid: "0422aead5624494d839898bb6d2b4db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd61df92fc24247b56af0b50c5c477e" is_pre_election: true
I20260809 03:34:04.729305 11701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee941cd780d4796b24ed2fda2c45fc9" candidate_uuid: "0422aead5624494d839898bb6d2b4db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263a1203b74069a145188b22ad64db" is_pre_election: true
I20260809 03:34:04.729406 11894 raft_consensus.cc:2454] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0422aead5624494d839898bb6d2b4db7 in term 0.
I20260809 03:34:04.729575 11701 raft_consensus.cc:2454] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0422aead5624494d839898bb6d2b4db7 in term 0.
I20260809 03:34:04.729899 11418 leader_election.cc:302] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0422aead5624494d839898bb6d2b4db7, cdd61df92fc24247b56af0b50c5c477e; no voters:
I20260809 03:34:04.730172 12001 raft_consensus.cc:2781] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:04.730252 12001 raft_consensus.cc:482] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:04.730285 12001 raft_consensus.cc:3037] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:04.731287 12001 raft_consensus.cc:504] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.731601 12001 leader_election.cc:288] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [CANDIDATE]: Term 1 election: Requested vote from peers cdd61df92fc24247b56af0b50c5c477e (127.15.222.67:41347), 72263a1203b74069a145188b22ad64db (127.15.222.66:35147)
I20260809 03:34:04.731927 11701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee941cd780d4796b24ed2fda2c45fc9" candidate_uuid: "0422aead5624494d839898bb6d2b4db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263a1203b74069a145188b22ad64db"
I20260809 03:34:04.731943 11894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee941cd780d4796b24ed2fda2c45fc9" candidate_uuid: "0422aead5624494d839898bb6d2b4db7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd61df92fc24247b56af0b50c5c477e"
I20260809 03:34:04.732030 11701 raft_consensus.cc:3037] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:04.732048 11894 raft_consensus.cc:3037] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:04.733021 11701 raft_consensus.cc:2454] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0422aead5624494d839898bb6d2b4db7 in term 1.
I20260809 03:34:04.733026 11894 raft_consensus.cc:2454] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0422aead5624494d839898bb6d2b4db7 in term 1.
I20260809 03:34:04.733280 11418 leader_election.cc:302] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0422aead5624494d839898bb6d2b4db7, cdd61df92fc24247b56af0b50c5c477e; no voters:
I20260809 03:34:04.733477 12001 raft_consensus.cc:2781] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:04.733645 12001 raft_consensus.cc:686] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [term 1 LEADER]: Becoming Leader. State: Replica: 0422aead5624494d839898bb6d2b4db7, State: Running, Role: LEADER
I20260809 03:34:04.733975 12001 consensus_queue.cc:237] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } }
I20260809 03:34:04.736961 11304 catalog_manager.cc:5495] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0422aead5624494d839898bb6d2b4db7 (127.15.222.65). New cstate: current_term: 1 leader_uuid: "0422aead5624494d839898bb6d2b4db7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0422aead5624494d839898bb6d2b4db7" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 41925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 } health_report { overall_health: UNKNOWN } } }
I20260809 03:34:04.759267 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.756269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.15.222.65:41925)
0809 03:34:04.756370 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:04.756373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:04.756387 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:34:04.756525 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:04.756527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:04.756579 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:04.756659 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:04.756820 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.757299 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.758079 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:04.758283 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:04.758350 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:04.758587 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:04.758988 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:04.759070 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:04.759097 (+ 27us) server_negotiation.cc:299] Negotiation successful
0809 03:34:04.759139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:34:04.764807 11894 raft_consensus.cc:1261] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 1 FOLLOWER]: Refusing update from remote peer 0422aead5624494d839898bb6d2b4db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:04.764822 11701 raft_consensus.cc:1261] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 1 FOLLOWER]: Refusing update from remote peer 0422aead5624494d839898bb6d2b4db7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:04.765477 12001 consensus_queue.cc:1035] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263a1203b74069a145188b22ad64db" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:04.765652 12008 consensus_queue.cc:1035] T dee941cd780d4796b24ed2fda2c45fc9 P 0422aead5624494d839898bb6d2b4db7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd61df92fc24247b56af0b50c5c477e" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:04.772047 12014 mvcc.cc:205] Tried to move back new op lower bound from 7316465437749698560 to 7316465437632892928. Current Snapshot: MvccSnapshot[applied={T|T < 7316465437749698560}]
I20260809 03:34:04.772495 12016 mvcc.cc:205] Tried to move back new op lower bound from 7316465437749698560 to 7316465437632892928. Current Snapshot: MvccSnapshot[applied={T|T < 7316465437749698560}]
I20260809 03:34:04.781342 11303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:04.781519 11303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:04.783917 11303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b425cd5f9f6403daaf452116a038691: Sending DeleteTablet for 3 replicas of tablet dee941cd780d4796b24ed2fda2c45fc9
I20260809 03:34:04.784650 11674 tablet_service.cc:1514] Processing DeleteTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:04 UTC) from {username='slave'} at 127.0.0.1:39062
I20260809 03:34:04.784718 11863 tablet_service.cc:1514] Processing DeleteTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:04 UTC) from {username='slave'} at 127.0.0.1:37270
I20260809 03:34:04.784834 11487 tablet_service.cc:1514] Processing DeleteTablet for tablet dee941cd780d4796b24ed2fda2c45fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:04 UTC) from {username='slave'} at 127.0.0.1:34162
I20260809 03:34:04.785243 12033 tablet_replica.cc:330] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: stopping tablet replica
I20260809 03:34:04.785256 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 11391
I20260809 03:34:04.785522 12033 raft_consensus.cc:2229] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:04.787943 12033 raft_consensus.cc:2258] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:04.793345 12033 ts_tablet_manager.cc:1905] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:04.795140 12032 tablet_replica.cc:330] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: stopping tablet replica
I20260809 03:34:04.796229 12033 ts_tablet_manager.cc:1918] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:04.796360 12033 log.cc:1196] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/dee941cd780d4796b24ed2fda2c45fc9
I20260809 03:34:04.796522 12032 raft_consensus.cc:2229] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:04.796747 12033 ts_tablet_manager.cc:1939] T dee941cd780d4796b24ed2fda2c45fc9 P cdd61df92fc24247b56af0b50c5c477e: Deleting consensus metadata
I20260809 03:34:04.796878 12032 raft_consensus.cc:2258] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:04.798312 11285 catalog_manager.cc:4841] TS cdd61df92fc24247b56af0b50c5c477e (127.15.222.67:41347): tablet dee941cd780d4796b24ed2fda2c45fc9 (table test-table [id=f3d94c7bc6d74e508da085f576b219fb]) successfully deleted
I20260809 03:34:04.799013 12032 ts_tablet_manager.cc:1905] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:04.800981 12032 ts_tablet_manager.cc:1918] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:04.801064 12032 log.cc:1196] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/dee941cd780d4796b24ed2fda2c45fc9
I20260809 03:34:04.801293 12032 ts_tablet_manager.cc:1939] T dee941cd780d4796b24ed2fda2c45fc9 P 72263a1203b74069a145188b22ad64db: Deleting consensus metadata
I20260809 03:34:04.802295 11287 catalog_manager.cc:4841] TS 72263a1203b74069a145188b22ad64db (127.15.222.66:35147): tablet dee941cd780d4796b24ed2fda2c45fc9 (table test-table [id=f3d94c7bc6d74e508da085f576b219fb]) successfully deleted
W20260809 03:34:04.803746 11285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:41925 (error 108)
I20260809 03:34:04.803984 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 11583
I20260809 03:34:04.807403 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:04.807106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:41925 (local address 127.0.0.1:34172)
0809 03:34:04.807247 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:04.807322 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 03:34:04.807569 11285 catalog_manager.cc:4584] TS 0422aead5624494d839898bb6d2b4db7 (127.15.222.65:41925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dee941cd780d4796b24ed2fda2c45fc9: Network error: Client connection negotiation failed: client connection to 127.15.222.65:41925: connect: Connection refused (error 111)
I20260809 03:34:04.812639 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 11773
I20260809 03:34:04.820736 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 11256
2026-08-09T03:34:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:04 dist-test-slave-cvwc krb5kdc[12041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:04 dist-test-slave-cvwc krb5kdc[12041](info): set up 2 sockets
Aug 09 03:34:04 dist-test-slave-cvwc krb5kdc[12041](info): commencing operation
krb5kdc: starting...
W20260809 03:34:05.970418 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:06.085990 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41373
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:06.195290 12062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:06.195538 12062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:06.195578 12062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:06.199468 12062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:06.199517 12062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:06.199535 12062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:06.199566 12062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:06.199592 12062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:06.204586 12062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:06.205601 12062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:06.206614 12062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:06.214636 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.214802 12062 server_base.cc:1034] running on GCE node
W20260809 03:34:06.214704 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:06.214704 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.215538 12062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:06.216557 12062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:06.217732 12062 hybrid_clock.cc:648] HybridClock initialized: now 1786246446217696 us; error 26 us; skew 500 ppm
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:06.220278 12062 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:06.221107 12062 webserver.cc:460] Webserver started at http://127.15.222.126:42491/ using document root <none> and password file <none>
I20260809 03:34:06.221596 12062 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:06.221649 12062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:06.221854 12062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:06.223773 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "9894a0f7834b4ebd947c645502191d3c"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ccb2c41538491b3df53f3ae419a93fd"
server_key_iv: "4afcfa1f7b9dff08da5ae9bffbf00825"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.224277 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "9894a0f7834b4ebd947c645502191d3c"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ccb2c41538491b3df53f3ae419a93fd"
server_key_iv: "4afcfa1f7b9dff08da5ae9bffbf00825"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.227677 12062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:06.229839 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.230947 12062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:06.231057 12062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "9894a0f7834b4ebd947c645502191d3c"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ccb2c41538491b3df53f3ae419a93fd"
server_key_iv: "4afcfa1f7b9dff08da5ae9bffbf00825"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.231144 12062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:06.262552 12062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:06.265169 12062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:06.265336 12062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:06.271864 12062 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41373
I20260809 03:34:06.271880 12165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41373 every 8 connection(s)
I20260809 03:34:06.272775 12062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:06.275701 12166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:06.281100 12166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: Bootstrap starting.
I20260809 03:34:06.281680 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 12062
I20260809 03:34:06.281829 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:06.282091 16249 external_mini_cluster.cc:1428] Setting key 06e1066b79aebb99f579d9846bb0b9d7
I20260809 03:34:06.283697 12166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:06.284382 12166 log.cc:824] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:06.286480 12166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: No bootstrap required, opened a new log
I20260809 03:34:06.290576 12166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } }
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:06.290781 12166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:06.290824 12166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9894a0f7834b4ebd947c645502191d3c, State: Initialized, Role: FOLLOWER
I20260809 03:34:06.291234 12166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } }
I20260809 03:34:06.291359 12166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:06.291424 12166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:06.291519 12166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:06.292294 12166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } }
I20260809 03:34:06.292604 12166 leader_election.cc:302] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9894a0f7834b4ebd947c645502191d3c; no voters:
I20260809 03:34:06.292902 12166 leader_election.cc:288] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:06.292963 12173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:06.293157 12173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [term 1 LEADER]: Becoming Leader. State: Replica: 9894a0f7834b4ebd947c645502191d3c, State: Running, Role: LEADER
I20260809 03:34:06.293460 12173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } }
I20260809 03:34:06.294178 12166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:06.294626 12173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9894a0f7834b4ebd947c645502191d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } } }
I20260809 03:34:06.295166 12173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:06.295719 12181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9894a0f7834b4ebd947c645502191d3c. Latest consensus state: current_term: 1 leader_uuid: "9894a0f7834b4ebd947c645502191d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9894a0f7834b4ebd947c645502191d3c" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41373 } } }
I20260809 03:34:06.295823 12181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:06.296527 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.283318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.15.222.126:41373)
0809 03:34:06.283706 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:06.283721 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:06.283759 (+ 38us) server_negotiation.cc:407] Connection header received
0809 03:34:06.284374 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:06.284401 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:06.284753 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:06.285115 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:06.285945 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.286925 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.288109 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.288555 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.288660 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.291307 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:06.291312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:06.291324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:06.291349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:06.293086 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.293526 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.293533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.293539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.293767 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.294017 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.294021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.294023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.294440 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:06.295194 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:06.295337 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 03:34:06.295621 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260809 03:34:06.296836 12191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:06.300186 12191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:06.305514 12191 catalog_manager.cc:1347] Generated new cluster ID: 9f6966b7eede4f6e8b6178d0d8653597
I20260809 03:34:06.305570 12191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:06.321833 12191 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:06.322676 12191 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:06.329664 12191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: Generated new TSK 0
I20260809 03:34:06.330283 12191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:34:06.357565 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:06.468086 12209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:06.468401 12209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:06.468474 12209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:06.473003 12209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:06.473064 12209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:06.473134 12209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:06.478710 12209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:06.479686 12209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:06.480680 12209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:06.490082 12223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.490144 12209 server_base.cc:1034] running on GCE node
W20260809 03:34:06.490074 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:06.490078 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.490694 12209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:06.491276 12209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:06.492435 12209 hybrid_clock.cc:648] HybridClock initialized: now 1786246446492417 us; error 29 us; skew 500 ppm
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:06.498313 12209 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:34:06.499217 12209 webserver.cc:460] Webserver started at http://127.15.222.65:34217/ using document root <none> and password file <none>
I20260809 03:34:06.499763 12209 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:06.499830 12209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:06.500020 12209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:06.502064 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "7bfa72c27d3d4986ad18d1ca5615a873"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "3a7076890bfb35e6f1d34f6def6ee4d8"
server_key_iv: "345f4e82c29aa7984fa230eb852c44cb"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.502635 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "7bfa72c27d3d4986ad18d1ca5615a873"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "3a7076890bfb35e6f1d34f6def6ee4d8"
server_key_iv: "345f4e82c29aa7984fa230eb852c44cb"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.506420 12209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 03:34:06.508630 12233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.509758 12209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 03:34:06.509871 12209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "7bfa72c27d3d4986ad18d1ca5615a873"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "3a7076890bfb35e6f1d34f6def6ee4d8"
server_key_iv: "345f4e82c29aa7984fa230eb852c44cb"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.509974 12209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:06.522637 12209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:06.525182 12209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:06.525339 12209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:06.525898 12209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:06.526659 12209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:06.526703 12209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.526753 12209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:06.526782 12209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.534775 12209 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:37875
I20260809 03:34:06.534804 12419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:37875 every 8 connection(s)
I20260809 03:34:06.535626 12209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:06.544154 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 12209
I20260809 03:34:06.544263 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:06.544554 16249 external_mini_cluster.cc:1428] Setting key 105a5ca321d11fccdbf96547c544cef2
I20260809 03:34:06.548638 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.537285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:54701 (local address 127.15.222.126:41373)
0809 03:34:06.537440 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:06.537445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:06.538230 (+ 785us) server_negotiation.cc:407] Connection header received
0809 03:34:06.539495 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:06.539499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:06.539555 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:06.539626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:06.540727 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.541252 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.542388 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.542619 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.542708 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.545047 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:06.545050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:06.545052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:06.545083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:06.546679 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.547156 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.547160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.547162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.547248 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.547558 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.547561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.547563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.547766 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:06.547857 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:06.548358 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 03:34:06.548481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 03:34:06.549350 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.537581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41373 (local address 127.15.222.65:54701)
0809 03:34:06.538081 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:06.538122 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:06.539280 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:06.539800 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:06.539807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:06.540124 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:06.540579 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.540592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.541368 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.541371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.542272 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.542282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.542753 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.542757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.542911 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.543420 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:06.543436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:06.544864 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:06.546798 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.546802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.546814 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.547064 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.547332 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.547334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.547336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.547484 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.547850 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:06.547855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:06.548140 (+ 285us) client_negotiation.cc:769] Sending connection context
0809 03:34:06.548292 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 03:34:06.548537 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":145,"threads_started":1}
I20260809 03:34:06.550495 12420 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41373
I20260809 03:34:06.550746 12420 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:06.551335 12420 heartbeater.cc:511] Master 127.15.222.126:41373 requested a full tablet report, sending...
I20260809 03:34:06.552858 12114 ts_manager.cc:194] Registered new tserver with Master: 7bfa72c27d3d4986ad18d1ca5615a873 (127.15.222.65:37875)
I20260809 03:34:06.554033 12114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:54701
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:34:06.571807 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:06.683282 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:06.683538 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:06.683579 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:06.687538 12426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:06.687589 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:06.687645 12426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:06.692727 12426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:06.693817 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:06.694846 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:06.704165 12434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.704345 12426 server_base.cc:1034] running on GCE node
W20260809 03:34:06.704151 12437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:06.704170 12435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.704900 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:06.705492 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:06.706648 12426 hybrid_clock.cc:648] HybridClock initialized: now 1786246446706628 us; error 36 us; skew 500 ppm
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:06.709143 12426 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:34:06.709987 12426 webserver.cc:460] Webserver started at http://127.15.222.66:45715/ using document root <none> and password file <none>
I20260809 03:34:06.710507 12426 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:06.710572 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:06.710767 12426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:06.712761 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "dfa4f396562c413c93e90c9e9076ac49"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "5786d3e5caf8c804bf01e270559790e2"
server_key_iv: "70538c864c7eed78dca9831070cda0fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.713301 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "dfa4f396562c413c93e90c9e9076ac49"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "5786d3e5caf8c804bf01e270559790e2"
server_key_iv: "70538c864c7eed78dca9831070cda0fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.716821 12426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:06.719090 12449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.720208 12426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:06.720312 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "dfa4f396562c413c93e90c9e9076ac49"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "5786d3e5caf8c804bf01e270559790e2"
server_key_iv: "70538c864c7eed78dca9831070cda0fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.720399 12426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:06.733891 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:06.736469 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:06.736627 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:06.737147 12426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:06.737924 12426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:06.737970 12426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.738020 12426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:06.738047 12426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.746263 12426 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:43591
I20260809 03:34:06.746285 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:43591 every 8 connection(s)
I20260809 03:34:06.747094 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:34:06.747844 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 12426
I20260809 03:34:06.747978 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:06.748296 16249 external_mini_cluster.cc:1428] Setting key 7dacf9cfe0d2e22e952bc85a7fbdbac8
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
I20260809 03:34:06.761214 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.748928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:42171 (local address 127.15.222.126:41373)
0809 03:34:06.749064 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:06.749069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:06.750036 (+ 967us) server_negotiation.cc:407] Connection header received
0809 03:34:06.751651 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:06.751654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:06.751702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:06.751771 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:06.753013 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.753786 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.754924 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.755266 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.755375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.757769 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:06.757773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:06.757776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:06.757805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:06.759276 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.759718 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.759720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.759722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.759792 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.760079 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.760081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.760082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.760251 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:06.760321 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:06.760900 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 03:34:06.761045 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
I20260809 03:34:06.761986 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.749340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41373 (local address 127.15.222.66:42171)
0809 03:34:06.749864 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:06.749929 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:06.751413 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:06.751967 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:06.751978 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:06.752377 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:06.752849 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.752862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.753912 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.753916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.754795 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.754805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.755408 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.755412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.755563 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.756116 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:06.756133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:06.757562 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:06.759372 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.759376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.759390 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.759636 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.759863 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.759865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.759866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.760012 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.760323 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:06.760328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:06.760581 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 03:34:06.760754 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 03:34:06.761030 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260809 03:34:06.763290 12628 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41373
I20260809 03:34:06.763530 12628 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:06.764025 12628 heartbeater.cc:511] Master 127.15.222.126:41373 requested a full tablet report, sending...
I20260809 03:34:06.765149 12114 ts_manager.cc:194] Registered new tserver with Master: dfa4f396562c413c93e90c9e9076ac49 (127.15.222.66:43591)
I20260809 03:34:06.765741 12114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:42171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:34:06.776284 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:06.889591 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:06.889854 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:06.889910 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:06.893869 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:06.893942 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:06.894027 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:06.899282 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:06.900316 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:06.901326 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:06.910634 12646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:06.910676 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:06.910751 12652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:06.910717 12637 server_base.cc:1034] running on GCE node
I20260809 03:34:06.911274 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:06.911876 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:06.913029 12637 hybrid_clock.cc:648] HybridClock initialized: now 1786246446913011 us; error 30 us; skew 500 ppm
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:06.915661 12637 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:34:06.916481 12637 webserver.cc:460] Webserver started at http://127.15.222.67:41435/ using document root <none> and password file <none>
I20260809 03:34:06.917006 12637 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:06.917073 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:06.917279 12637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:06.919329 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "2771e0c8822048ac82528e4aefaf25ef"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ee9c3ddf0ab14da4bd7e6e725a53ef6"
server_key_iv: "7ee5e124a88a8638ab4a449c2c5532c9"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.919855 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "2771e0c8822048ac82528e4aefaf25ef"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ee9c3ddf0ab14da4bd7e6e725a53ef6"
server_key_iv: "7ee5e124a88a8638ab4a449c2c5532c9"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.923414 12637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 03:34:06.925812 12662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.927033 12637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 03:34:06.927145 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "2771e0c8822048ac82528e4aefaf25ef"
format_stamp: "Formatted at 2026-08-09 03:34:06 on dist-test-slave-cvwc"
server_key: "2ee9c3ddf0ab14da4bd7e6e725a53ef6"
server_key_iv: "7ee5e124a88a8638ab4a449c2c5532c9"
server_key_version: "encryptionkey@0"
I20260809 03:34:06.927263 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:06.939220 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:06.941915 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:06.942077 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:06.942616 12637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:06.943468 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:06.943516 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.943568 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:06.943595 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:06.951844 12637 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:36909
I20260809 03:34:06.951911 12844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:36909 every 8 connection(s)
I20260809 03:34:06.952700 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:06.953163 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 12637
I20260809 03:34:06.953280 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:06.953585 16249 external_mini_cluster.cc:1428] Setting key 04c3e9f7da813ef061fdcccd0f8f14dc
Aug 09 03:34:06 dist-test-slave-cvwc krb5kdc[12041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246446, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:06.965214 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.954563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:35295 (local address 127.15.222.126:41373)
0809 03:34:06.954684 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:06.954689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:06.955747 (+ 1058us) server_negotiation.cc:407] Connection header received
0809 03:34:06.957071 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:06.957073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:06.957118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:06.957173 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:06.958283 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.958819 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.959851 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.960050 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.960128 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.962383 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:06.962386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:06.962388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:06.962419 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:06.963673 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.964065 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.964067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.964069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.964135 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.964368 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.964370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.964371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.964536 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:06.964603 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:06.965002 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 03:34:06.965094 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:06.966043 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.955042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41373 (local address 127.15.222.67:35295)
0809 03:34:06.955589 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:06.955642 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:06.956872 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:06.957310 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:06.957316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:06.957645 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:06.958179 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.958192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.958914 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.958916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.959763 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:06.959772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.960158 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.960161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:06.960297 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.960776 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:06.960791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:06.962232 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:06.963753 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.963756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.963766 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.963993 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.964193 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:06.964195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:06.964196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:06.964308 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:06.964602 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:06.964606 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:06.964819 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 03:34:06.964965 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 03:34:06.965197 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":144,"threads_started":1}
I20260809 03:34:06.967140 12845 heartbeater.cc:347] Connected to a master server at 127.15.222.126:41373
I20260809 03:34:06.967363 12845 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:06.967824 12845 heartbeater.cc:511] Master 127.15.222.126:41373 requested a full tablet report, sending...
I20260809 03:34:06.968757 12114 ts_manager.cc:194] Registered new tserver with Master: 2771e0c8822048ac82528e4aefaf25ef (127.15.222.67:36909)
I20260809 03:34:06.969309 12114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:35295
I20260809 03:34:06.979327 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:06.987780 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:06.981179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.15.222.126:41373)
0809 03:34:06.981332 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:06.981337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:06.981448 (+ 111us) server_negotiation.cc:407] Connection header received
0809 03:34:06.981543 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:06.981545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:06.981584 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:06.981648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:06.982345 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.982844 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.983571 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:06.983762 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:06.983825 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:06.985184 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:06.985187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:06.985189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:06.985214 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:06.986346 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.986778 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.986780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.986781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.986847 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:06.987068 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:06.987070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:06.987071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:06.987258 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:06.987361 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:06.987547 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 03:34:06.987620 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 03:34:06.990613 12114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:06.993513 12114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:07.007380 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.002164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.15.222.66:43591)
0809 03:34:07.002568 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.002580 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.002603 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:34:07.002680 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.002686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.002913 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.003072 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:07.003527 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.004387 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.006020 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.006711 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.006959 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.007110 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.007161 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.007247 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":101,"threads_started":1}
I20260809 03:34:07.007372 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.001689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:43591 (local address 127.0.0.1:34946)
0809 03:34:07.002400 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.002415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.002503 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.003223 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.003229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.003253 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:07.003402 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.003410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.004506 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.004510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.005621 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.005632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.006833 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.006836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.006959 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.007008 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.007104 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.007222 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":48,"thread_start_us":133,"threads_started":1}
I20260809 03:34:07.007768 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.001574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:36909 (local address 127.0.0.1:54646)
0809 03:34:07.002231 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.002279 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.002403 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.003563 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.003565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.003588 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:07.003692 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.003699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.005144 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.005147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.006287 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.006298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.007343 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.007347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.007469 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.007511 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.007570 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.007637 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":70,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260809 03:34:07.008247 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.002300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37875 (local address 127.0.0.1:40340)
0809 03:34:07.002775 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.002788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.002873 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.003553 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.003556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.003584 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:07.003770 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.003781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.005180 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.005183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.006240 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.006250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.007343 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.007347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.007472 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.007994 (+ 522us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.008048 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.008120 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":581,"spinlock_wait_cycles":10752,"thread_start_us":101,"threads_started":1}
I20260809 03:34:07.008419 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.002721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.15.222.65:37875)
0809 03:34:07.003117 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.003128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.003150 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:34:07.003222 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.003227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.003411 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.003707 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:07.003880 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.005003 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.006355 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.007202 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.007535 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.007678 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.008100 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.008198 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260809 03:34:07.008837 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.001744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.15.222.67:36909)
0809 03:34:07.003154 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.003167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.003190 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:34:07.003257 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.003261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.003410 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.003580 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:07.003826 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.005003 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.006389 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.007202 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.008404 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.008561 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.008610 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.008707 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":168,"threads_started":1}
I20260809 03:34:07.010892 12522 tablet_service.cc:1467] Processing CreateTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d (DEFAULT_TABLE table=test-table [id=c8eb18e6606d4c34a0460d6ced39c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:07.011370 12306 tablet_service.cc:1467] Processing CreateTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d (DEFAULT_TABLE table=test-table [id=c8eb18e6606d4c34a0460d6ced39c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:07.012097 12522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:07.011788 12736 tablet_service.cc:1467] Processing CreateTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d (DEFAULT_TABLE table=test-table [id=c8eb18e6606d4c34a0460d6ced39c3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:07.012399 12306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:07.012634 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:07.018503 12873 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Bootstrap starting.
I20260809 03:34:07.019775 12874 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Bootstrap starting.
I20260809 03:34:07.020170 12876 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Bootstrap starting.
I20260809 03:34:07.021170 12873 tablet_bootstrap.cc:654] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:07.022282 12876 tablet_bootstrap.cc:654] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:07.022483 12873 log.cc:824] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:07.023449 12876 log.cc:824] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:07.023669 12874 tablet_bootstrap.cc:654] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:07.024801 12873 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: No bootstrap required, opened a new log
I20260809 03:34:07.025007 12873 ts_tablet_manager.cc:1397] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260809 03:34:07.024982 12874 log.cc:824] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:07.025099 12876 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: No bootstrap required, opened a new log
I20260809 03:34:07.025272 12876 ts_tablet_manager.cc:1397] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:34:07.026971 12874 tablet_bootstrap.cc:492] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: No bootstrap required, opened a new log
I20260809 03:34:07.027204 12874 ts_tablet_manager.cc:1397] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260809 03:34:07.028496 12876 raft_consensus.cc:348] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.028713 12876 raft_consensus.cc:374] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:07.028784 12876 raft_consensus.cc:729] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa4f396562c413c93e90c9e9076ac49, State: Initialized, Role: FOLLOWER
I20260809 03:34:07.028764 12873 raft_consensus.cc:348] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.029112 12873 raft_consensus.cc:374] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:07.029181 12873 raft_consensus.cc:729] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2771e0c8822048ac82528e4aefaf25ef, State: Initialized, Role: FOLLOWER
I20260809 03:34:07.029215 12876 consensus_queue.cc:260] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.029675 12873 consensus_queue.cc:260] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.029892 12628 heartbeater.cc:503] Master 127.15.222.126:41373 was elected leader, sending a full tablet report...
I20260809 03:34:07.030680 12873 ts_tablet_manager.cc:1428] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:34:07.030874 12874 raft_consensus.cc:348] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.031065 12845 heartbeater.cc:503] Master 127.15.222.126:41373 was elected leader, sending a full tablet report...
I20260809 03:34:07.031111 12876 ts_tablet_manager.cc:1428] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:34:07.031153 12874 raft_consensus.cc:374] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:07.031209 12874 raft_consensus.cc:729] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfa72c27d3d4986ad18d1ca5615a873, State: Initialized, Role: FOLLOWER
I20260809 03:34:07.031610 12874 consensus_queue.cc:260] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.032498 12874 ts_tablet_manager.cc:1428] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:34:07.032819 12420 heartbeater.cc:503] Master 127.15.222.126:41373 was elected leader, sending a full tablet report...
W20260809 03:34:07.036609 12422 tablet.cc:2369] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:07.163821 12884 raft_consensus.cc:482] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:07.164031 12884 raft_consensus.cc:504] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.165180 12884 leader_election.cc:288] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2771e0c8822048ac82528e4aefaf25ef (127.15.222.67:36909), 7bfa72c27d3d4986ad18d1ca5615a873 (127.15.222.65:37875)
I20260809 03:34:07.169742 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.165589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:52759 (local address 127.15.222.65:37875)
0809 03:34:07.165744 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.165748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.165927 (+ 179us) server_negotiation.cc:407] Connection header received
0809 03:34:07.166022 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.166026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.166075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.166143 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:07.166385 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.167092 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.168519 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.169113 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.169230 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.169283 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.169518 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.169585 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 03:34:07.169742 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.165509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:53133 (local address 127.15.222.67:36909)
0809 03:34:07.165622 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.165627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.165644 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:34:07.165839 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.165842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.165891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.165978 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:07.166238 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.166993 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.168524 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.169113 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.169230 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.169283 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.169518 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.169585 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:34:07.169764 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.165410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:36909 (local address 127.15.222.66:53133)
0809 03:34:07.165589 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.165605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.165691 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.166000 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.166003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.166019 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:07.166125 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.166132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.167093 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.167096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.168404 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.168417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.169223 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.169226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.169384 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.169422 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.169495 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.169586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":51}
I20260809 03:34:07.169775 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.165435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37875 (local address 127.15.222.66:52759)
0809 03:34:07.165843 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.165853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.165936 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.166169 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.166172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.166188 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:07.166288 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.166293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.167184 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.167187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.168417 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.168423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.169223 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.169226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.169367 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.169434 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.169495 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.169586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":132,"thread_start_us":145,"threads_started":1}
I20260809 03:34:07.170483 12765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ef6fc8fa954c13b8d8c2b5962fdc2d" candidate_uuid: "dfa4f396562c413c93e90c9e9076ac49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2771e0c8822048ac82528e4aefaf25ef" is_pre_election: true
I20260809 03:34:07.170498 12335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ef6fc8fa954c13b8d8c2b5962fdc2d" candidate_uuid: "dfa4f396562c413c93e90c9e9076ac49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" is_pre_election: true
I20260809 03:34:07.170764 12335 raft_consensus.cc:2454] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa4f396562c413c93e90c9e9076ac49 in term 0.
I20260809 03:34:07.170768 12765 raft_consensus.cc:2454] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa4f396562c413c93e90c9e9076ac49 in term 0.
I20260809 03:34:07.171242 12451 leader_election.cc:302] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2771e0c8822048ac82528e4aefaf25ef, 7bfa72c27d3d4986ad18d1ca5615a873, dfa4f396562c413c93e90c9e9076ac49; no voters:
I20260809 03:34:07.171485 12884 raft_consensus.cc:2781] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:07.171559 12884 raft_consensus.cc:482] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:07.171607 12884 raft_consensus.cc:3037] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:07.172626 12884 raft_consensus.cc:504] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.172961 12884 leader_election.cc:288] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [CANDIDATE]: Term 1 election: Requested vote from peers 2771e0c8822048ac82528e4aefaf25ef (127.15.222.67:36909), 7bfa72c27d3d4986ad18d1ca5615a873 (127.15.222.65:37875)
I20260809 03:34:07.173282 12765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ef6fc8fa954c13b8d8c2b5962fdc2d" candidate_uuid: "dfa4f396562c413c93e90c9e9076ac49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2771e0c8822048ac82528e4aefaf25ef"
I20260809 03:34:07.173290 12335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ef6fc8fa954c13b8d8c2b5962fdc2d" candidate_uuid: "dfa4f396562c413c93e90c9e9076ac49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfa72c27d3d4986ad18d1ca5615a873"
I20260809 03:34:07.173388 12765 raft_consensus.cc:3037] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:07.173394 12335 raft_consensus.cc:3037] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:07.174443 12335 raft_consensus.cc:2454] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa4f396562c413c93e90c9e9076ac49 in term 1.
I20260809 03:34:07.174484 12765 raft_consensus.cc:2454] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa4f396562c413c93e90c9e9076ac49 in term 1.
I20260809 03:34:07.174746 12451 leader_election.cc:302] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2771e0c8822048ac82528e4aefaf25ef, 7bfa72c27d3d4986ad18d1ca5615a873, dfa4f396562c413c93e90c9e9076ac49; no voters:
I20260809 03:34:07.174968 12884 raft_consensus.cc:2781] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:07.175155 12884 raft_consensus.cc:686] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 1 LEADER]: Becoming Leader. State: Replica: dfa4f396562c413c93e90c9e9076ac49, State: Running, Role: LEADER
I20260809 03:34:07.175473 12884 consensus_queue.cc:237] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } }
I20260809 03:34:07.178550 12110 catalog_manager.cc:5495] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa4f396562c413c93e90c9e9076ac49 (127.15.222.66). New cstate: current_term: 1 leader_uuid: "dfa4f396562c413c93e90c9e9076ac49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa4f396562c413c93e90c9e9076ac49" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 43591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 } health_report { overall_health: UNKNOWN } } }
W20260809 03:34:07.203648 12846 tablet.cc:2369] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:07.204103 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.200948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.15.222.66:43591)
0809 03:34:07.201059 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.201062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.201075 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:07.201180 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.201182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.201234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.201311 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:07.201509 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.202023 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.202772 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.202980 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.203047 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.203269 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:07.203659 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:07.203772 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.203902 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.203940 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:34:07.209633 12765 raft_consensus.cc:1261] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 1 FOLLOWER]: Refusing update from remote peer dfa4f396562c413c93e90c9e9076ac49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:07.209635 12335 raft_consensus.cc:1261] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 7bfa72c27d3d4986ad18d1ca5615a873 [term 1 FOLLOWER]: Refusing update from remote peer dfa4f396562c413c93e90c9e9076ac49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:07.210309 12884 consensus_queue.cc:1035] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfa72c27d3d4986ad18d1ca5615a873" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:07.210494 12888 consensus_queue.cc:1035] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2771e0c8822048ac82528e4aefaf25ef" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:07.219511 12897 mvcc.cc:205] Tried to move back new op lower bound from 7316465447764054016 to 7316465447633268736. Current Snapshot: MvccSnapshot[applied={T|T < 7316465447764054016}]
I20260809 03:34:07.221856 12898 mvcc.cc:205] Tried to move back new op lower bound from 7316465447764054016 to 7316465447633268736. Current Snapshot: MvccSnapshot[applied={T|T < 7316465447764054016 or (T in {7316465447764054016})}]
I20260809 03:34:07.225561 12110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:07.225692 12110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:07.227843 12110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9894a0f7834b4ebd947c645502191d3c: Sending DeleteTablet for 3 replicas of tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d
I20260809 03:34:07.228608 12522 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:34946
I20260809 03:34:07.228624 12736 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:54646
I20260809 03:34:07.228621 12306 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:07 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:40340
I20260809 03:34:07.229075 12910 tablet_replica.cc:330] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: stopping tablet replica
I20260809 03:34:07.229360 12910 raft_consensus.cc:2229] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:07.229568 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 12209
I20260809 03:34:07.229619 12911 tablet_replica.cc:330] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: stopping tablet replica
I20260809 03:34:07.230634 12911 raft_consensus.cc:2229] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:34:07.231362 12910 raft_consensus.cc:2258] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:07.230988 12911 raft_consensus.cc:2258] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:07.237831 12911 ts_tablet_manager.cc:1905] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:07.241024 12911 ts_tablet_manager.cc:1918] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:07.241117 12911 log.cc:1196] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/a0ef6fc8fa954c13b8d8c2b5962fdc2d
I20260809 03:34:07.241372 12911 ts_tablet_manager.cc:1939] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P dfa4f396562c413c93e90c9e9076ac49: Deleting consensus metadata
I20260809 03:34:07.242486 12092 catalog_manager.cc:4841] TS dfa4f396562c413c93e90c9e9076ac49 (127.15.222.66:43591): tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d (table test-table [id=c8eb18e6606d4c34a0460d6ced39c3c0]) successfully deleted
I20260809 03:34:07.243706 12910 ts_tablet_manager.cc:1905] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:07.245662 12910 ts_tablet_manager.cc:1918] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:07.245774 12910 log.cc:1196] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/a0ef6fc8fa954c13b8d8c2b5962fdc2d
I20260809 03:34:07.246042 12910 ts_tablet_manager.cc:1939] T a0ef6fc8fa954c13b8d8c2b5962fdc2d P 2771e0c8822048ac82528e4aefaf25ef: Deleting consensus metadata
I20260809 03:34:07.247045 12093 catalog_manager.cc:4841] TS 2771e0c8822048ac82528e4aefaf25ef (127.15.222.67:36909): tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d (table test-table [id=c8eb18e6606d4c34a0460d6ced39c3c0]) successfully deleted
W20260809 03:34:07.249608 12092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:37875 (error 108)
I20260809 03:34:07.250474 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 12426
I20260809 03:34:07.251169 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.250322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:37875 (local address 127.0.0.1:40348)
0809 03:34:07.250996 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.251079 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
W20260809 03:34:07.259642 12092 catalog_manager.cc:4584] TS 7bfa72c27d3d4986ad18d1ca5615a873 (127.15.222.65:37875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0ef6fc8fa954c13b8d8c2b5962fdc2d: Network error: Client connection negotiation failed: client connection to 127.15.222.65:37875: connect: Connection refused (error 111)
I20260809 03:34:07.264544 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 12637
I20260809 03:34:07.276998 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 12062
2026-08-09T03:34:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2460 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T03:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:07Z Disabled control of system clock
I20260809 03:34:07.316133 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:42431
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:42431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:07.425506 12918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:07.425778 12918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:07.425828 12918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:07.429793 12918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:07.429848 12918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:07.429868 12918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:07.429888 12918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:07.429908 12918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:07.434525 12918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:42431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:42431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:07.435505 12918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:07.436810 12918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:07.444926 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:07.444934 12927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:07.445068 12918 server_base.cc:1034] running on GCE node
W20260809 03:34:07.444942 12926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:07.445856 12918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:07.446897 12918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:07.448045 12918 hybrid_clock.cc:648] HybridClock initialized: now 1786246447448026 us; error 37 us; skew 500 ppm
I20260809 03:34:07.449704 12918 webserver.cc:460] Webserver started at http://127.15.222.126:38953/ using document root <none> and password file <none>
I20260809 03:34:07.450268 12918 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:07.450345 12918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:07.450537 12918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:07.452431 12918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3cfc4fdb70c847cf987263ced713df81"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd7189d8dad2c33b816f1d925e456a56"
server_key_iv: "0932f8ab9e48b8a5c1aa09f065155928"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.452920 12918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3cfc4fdb70c847cf987263ced713df81"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd7189d8dad2c33b816f1d925e456a56"
server_key_iv: "0932f8ab9e48b8a5c1aa09f065155928"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.456308 12918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260809 03:34:07.458659 12942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.459794 12918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:07.459898 12918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3cfc4fdb70c847cf987263ced713df81"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd7189d8dad2c33b816f1d925e456a56"
server_key_iv: "0932f8ab9e48b8a5c1aa09f065155928"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.459977 12918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:07.471251 12918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:07.471792 12918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:07.471933 12918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:07.478413 12918 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:42431
I20260809 03:34:07.478434 13013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:42431 every 8 connection(s)
I20260809 03:34:07.479338 12918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:07.480917 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 12918
I20260809 03:34:07.481019 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:07.481280 16249 external_mini_cluster.cc:1428] Setting key d75ba3f2f0f8e911ab4537b8746f407c
I20260809 03:34:07.482662 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:07.489449 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: Bootstrap starting.
I20260809 03:34:07.491303 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.482605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.15.222.126:42431)
0809 03:34:07.483270 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.483291 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.483334 (+ 43us) server_negotiation.cc:407] Connection header received
0809 03:34:07.484180 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.484202 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.484680 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.485045 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:07.485479 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.486936 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.487744 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.488434 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.488594 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.489302 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:07.489309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:07.489323 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:07.489821 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:07.489903 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:07.489930 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:07.490020 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.490138 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.490390 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260809 03:34:07.492093 13016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:07.492839 13016 log.cc:824] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:07.494671 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: No bootstrap required, opened a new log
I20260809 03:34:07.497959 13016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } }
I20260809 03:34:07.498205 13016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:07.498306 13016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfc4fdb70c847cf987263ced713df81, State: Initialized, Role: FOLLOWER
I20260809 03:34:07.498724 13016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } }
I20260809 03:34:07.498847 13016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:07.498907 13016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:07.499004 13016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:07.499816 13016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } }
I20260809 03:34:07.500131 13016 leader_election.cc:302] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cfc4fdb70c847cf987263ced713df81; no voters:
I20260809 03:34:07.500401 13016 leader_election.cc:288] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:07.500509 13022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:07.500696 13022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [term 1 LEADER]: Becoming Leader. State: Replica: 3cfc4fdb70c847cf987263ced713df81, State: Running, Role: LEADER
I20260809 03:34:07.501024 13022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } }
I20260809 03:34:07.501451 13016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:07.502563 13026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cfc4fdb70c847cf987263ced713df81. Latest consensus state: current_term: 1 leader_uuid: "3cfc4fdb70c847cf987263ced713df81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } } }
I20260809 03:34:07.502684 13026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:07.502950 13025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cfc4fdb70c847cf987263ced713df81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cfc4fdb70c847cf987263ced713df81" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42431 } } }
I20260809 03:34:07.503043 13025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81 [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:07.506220 13053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:07.506295 13053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:07.506425 13043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:07.507351 13043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:07.512418 13043 catalog_manager.cc:1347] Generated new cluster ID: abb8a73f0a574cd3aacf0f69f46e9f9a
I20260809 03:34:07.512476 13043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:07.523727 13043 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:07.524564 13043 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:07.531524 13043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: Generated new TSK 0
I20260809 03:34:07.532097 13043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:07.535943 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--builtin_ntp_servers=127.15.222.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:07.651576 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:07.651811 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:07.651850 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:07.655753 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:07.655802 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:07.655968 13059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:07.660879 13059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:07.661891 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:07.663357 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:07.672609 13072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:07.672613 13067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:07.672780 13059 server_base.cc:1034] running on GCE node
W20260809 03:34:07.672613 13068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:07.673241 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:07.673905 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:07.675058 13059 hybrid_clock.cc:648] HybridClock initialized: now 1786246447675041 us; error 24 us; skew 500 ppm
I20260809 03:34:07.676736 13059 webserver.cc:460] Webserver started at http://127.15.222.65:35117/ using document root <none> and password file <none>
I20260809 03:34:07.677251 13059 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:07.677307 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:07.677484 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:07.679508 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "e7de9c3bfe97408f8a314fdfa230fa9e"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "cbda3fc34ecd80bc5564a450d8bcaeee"
server_key_iv: "b2108260093a86ddf8e03bd8e014c38d"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.680006 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "e7de9c3bfe97408f8a314fdfa230fa9e"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "cbda3fc34ecd80bc5564a450d8bcaeee"
server_key_iv: "b2108260093a86ddf8e03bd8e014c38d"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.683631 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:07.685904 13084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.687049 13059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:07.687143 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "e7de9c3bfe97408f8a314fdfa230fa9e"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "cbda3fc34ecd80bc5564a450d8bcaeee"
server_key_iv: "b2108260093a86ddf8e03bd8e014c38d"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.687227 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:07.705948 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:07.706594 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:07.706741 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:07.707273 13059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:07.708209 13059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:07.708256 13059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.708297 13059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:07.708315 13059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.716398 13059 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:44339
I20260809 03:34:07.716432 13251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:44339 every 8 connection(s)
I20260809 03:34:07.717242 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:07.723484 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 13059
I20260809 03:34:07.723619 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:34:07.723932 16249 external_mini_cluster.cc:1428] Setting key e1f015e964e7aa967f4e8e7af29684c4
I20260809 03:34:07.726014 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--builtin_ntp_servers=127.15.222.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:07.726392 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.718966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:42597 (local address 127.15.222.126:42431)
0809 03:34:07.719093 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.719097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.720067 (+ 970us) server_negotiation.cc:407] Connection header received
0809 03:34:07.721361 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.721363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.721418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.721489 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:07.722329 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.722862 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.724289 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.724598 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.724695 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.725815 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:07.725818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:07.725820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:07.725832 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:07.725852 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:07.725860 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:07.725918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.726143 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.726248 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:34:07.727259 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.719321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42431 (local address 127.15.222.65:42597)
0809 03:34:07.719930 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.719969 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.721191 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.721638 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.721645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.721751 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:07.722225 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.722237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.722951 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.722954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.724130 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.724143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.724739 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.724741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.724880 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.725518 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:07.725520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:07.725569 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:07.725645 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:07.725941 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:07.725949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:07.725955 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.726083 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.726346 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260809 03:34:07.728466 13252 heartbeater.cc:347] Connected to a master server at 127.15.222.126:42431
I20260809 03:34:07.728699 13252 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:07.729255 13252 heartbeater.cc:511] Master 127.15.222.126:42431 requested a full tablet report, sending...
I20260809 03:34:07.730718 12963 ts_manager.cc:194] Registered new tserver with Master: e7de9c3bfe97408f8a314fdfa230fa9e (127.15.222.65:44339)
I20260809 03:34:07.731822 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.65:42597
W20260809 03:34:07.839246 13255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:07.839498 13255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:07.839560 13255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:07.843459 13255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:07.843514 13255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:07.843598 13255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:07.848227 13255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:07.849182 13255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:07.850550 13255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:07.859637 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:07.859645 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:07.859644 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:07.859782 13255 server_base.cc:1034] running on GCE node
I20260809 03:34:07.860320 13255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:07.860900 13255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:07.862052 13255 hybrid_clock.cc:648] HybridClock initialized: now 1786246447862037 us; error 25 us; skew 500 ppm
I20260809 03:34:07.863859 13255 webserver.cc:460] Webserver started at http://127.15.222.66:34141/ using document root <none> and password file <none>
I20260809 03:34:07.864384 13255 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:07.864442 13255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:07.864606 13255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:07.866618 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "d7c3c346df2c44d689ec8f10d9d4b19f"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd95abea0a7638e57116b4f14cfee7b9"
server_key_iv: "27ef80509c94c3ea5d79715d4abc8959"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.867129 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "d7c3c346df2c44d689ec8f10d9d4b19f"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd95abea0a7638e57116b4f14cfee7b9"
server_key_iv: "27ef80509c94c3ea5d79715d4abc8959"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.870604 13255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:07.872859 13276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.874107 13255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:07.874217 13255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "d7c3c346df2c44d689ec8f10d9d4b19f"
format_stamp: "Formatted at 2026-08-09 03:34:07 on dist-test-slave-cvwc"
server_key: "fd95abea0a7638e57116b4f14cfee7b9"
server_key_iv: "27ef80509c94c3ea5d79715d4abc8959"
server_key_version: "encryptionkey@0"
I20260809 03:34:07.874325 13255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:07.893906 13255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:07.894562 13255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:07.894730 13255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:07.895308 13255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:07.896236 13255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:07.896283 13255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.896337 13255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:07.896365 13255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:07.904719 13255 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42525
I20260809 03:34:07.904745 13435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42525 every 8 connection(s)
I20260809 03:34:07.905583 13255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:34:07.913127 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 13255
I20260809 03:34:07.913215 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:07.913486 16249 external_mini_cluster.cc:1428] Setting key d7bf81c0205c12cf5b3c9edb66d4cd93
I20260809 03:34:07.914021 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.907260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:43679 (local address 127.15.222.126:42431)
0809 03:34:07.907377 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:07.907380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:07.908376 (+ 996us) server_negotiation.cc:407] Connection header received
0809 03:34:07.909755 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:07.909758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:07.909814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:07.909885 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:07.910696 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.911212 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.912231 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.912429 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.912494 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.913513 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:07.913517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:07.913519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:07.913534 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:07.913559 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:07.913569 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:07.913632 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:07.913791 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 03:34:07.913862 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:34:07.914908 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:07.907609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42431 (local address 127.15.222.66:43679)
0809 03:34:07.908235 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:07.908274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:07.909544 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:07.910040 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:07.910047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:07.910126 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:07.910582 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.910595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.911301 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.911303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.912146 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:07.912155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:07.912530 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:07.912533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:07.912666 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:07.913238 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:07.913240 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:07.913293 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:07.913354 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:07.913634 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:07.913643 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:07.913649 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:34:07.913790 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 03:34:07.914058 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260809 03:34:07.915493 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--builtin_ntp_servers=127.15.222.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:07.915998 13437 heartbeater.cc:347] Connected to a master server at 127.15.222.126:42431
I20260809 03:34:07.916213 13437 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:07.916692 13437 heartbeater.cc:511] Master 127.15.222.126:42431 requested a full tablet report, sending...
I20260809 03:34:07.917644 12963 ts_manager.cc:194] Registered new tserver with Master: d7c3c346df2c44d689ec8f10d9d4b19f (127.15.222.66:42525)
I20260809 03:34:07.918208 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.66:43679
W20260809 03:34:08.027021 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:08.027252 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:08.027292 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:08.031159 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:08.031204 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:08.031260 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:08.035840 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:08.036751 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:08.038044 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:08.046878 13449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:08.046880 13448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:08.046960 13451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:08.047186 13440 server_base.cc:1034] running on GCE node
I20260809 03:34:08.047544 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:08.048108 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:08.049268 13440 hybrid_clock.cc:648] HybridClock initialized: now 1786246448049252 us; error 26 us; skew 500 ppm
I20260809 03:34:08.051045 13440 webserver.cc:460] Webserver started at http://127.15.222.67:42653/ using document root <none> and password file <none>
I20260809 03:34:08.051602 13440 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:08.051668 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:08.051869 13440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:08.053867 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "e7bd702471894a27a6d99deb7ecc19a2"
format_stamp: "Formatted at 2026-08-09 03:34:08 on dist-test-slave-cvwc"
server_key: "357e75dec88cc7c2f8c195837d1a4dfb"
server_key_iv: "227e3c861c8c039b78eef2116eef257a"
server_key_version: "encryptionkey@0"
I20260809 03:34:08.054404 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "e7bd702471894a27a6d99deb7ecc19a2"
format_stamp: "Formatted at 2026-08-09 03:34:08 on dist-test-slave-cvwc"
server_key: "357e75dec88cc7c2f8c195837d1a4dfb"
server_key_iv: "227e3c861c8c039b78eef2116eef257a"
server_key_version: "encryptionkey@0"
I20260809 03:34:08.057787 13440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:08.060003 13464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:08.061111 13440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:08.061213 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "e7bd702471894a27a6d99deb7ecc19a2"
format_stamp: "Formatted at 2026-08-09 03:34:08 on dist-test-slave-cvwc"
server_key: "357e75dec88cc7c2f8c195837d1a4dfb"
server_key_iv: "227e3c861c8c039b78eef2116eef257a"
server_key_version: "encryptionkey@0"
I20260809 03:34:08.061301 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:08.074863 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:08.075520 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:08.075680 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:08.076220 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:08.077147 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:08.077196 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:08.077248 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:08.077275 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:08.085388 13440 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:44491
I20260809 03:34:08.085439 13634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:44491 every 8 connection(s)
I20260809 03:34:08.086268 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:08.092072 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 13440
I20260809 03:34:08.092172 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:08.092414 16249 external_mini_cluster.cc:1428] Setting key 1f545ff4e2a6ede8d2ebbfa9573067d1
I20260809 03:34:08.094796 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.087925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:35781 (local address 127.15.222.126:42431)
0809 03:34:08.088050 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.088053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.089023 (+ 970us) server_negotiation.cc:407] Connection header received
0809 03:34:08.090421 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.090424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.090473 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.090534 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:08.091354 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.091864 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.093046 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.093263 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.093351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.094363 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:08.094367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:08.094369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:08.094380 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:08.094400 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:08.094407 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:08.094463 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.094601 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.094660 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:34:08.095561 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.088266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42431 (local address 127.15.222.67:35781)
0809 03:34:08.088896 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.088936 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.090205 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.090677 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.090683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.090762 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:08.091232 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.091245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.092002 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.092005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.092918 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.092928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.093414 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.093419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.093563 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.094126 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:08.094128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:08.094167 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:08.094227 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:08.094477 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:08.094485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:08.094491 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.094593 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.094852 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260809 03:34:08.096460 13635 heartbeater.cc:347] Connected to a master server at 127.15.222.126:42431
I20260809 03:34:08.096657 13635 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:08.097121 13635 heartbeater.cc:511] Master 127.15.222.126:42431 requested a full tablet report, sending...
I20260809 03:34:08.098026 12963 ts_manager.cc:194] Registered new tserver with Master: e7bd702471894a27a6d99deb7ecc19a2 (127.15.222.67:44491)
I20260809 03:34:08.098559 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.67:35781
I20260809 03:34:08.105818 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:08.111227 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.107623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.15.222.126:42431)
0809 03:34:08.107728 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.107731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.107883 (+ 152us) server_negotiation.cc:407] Connection header received
0809 03:34:08.107991 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.107993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.108031 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.108092 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:08.108278 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.108767 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.109620 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.109871 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.109963 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.110766 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:08.110770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:08.110772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:08.110783 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:08.110799 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:08.110807 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:08.110913 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.111063 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.111098 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:34:08.113675 12963 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:08.116545 12963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:08.129361 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.124105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44339 (local address 127.0.0.1:59170)
0809 03:34:08.124566 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.124613 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.124725 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.125453 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.125459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.125481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:08.125603 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.125609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.126870 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.126873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.127836 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.127847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.128868 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.128871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.128977 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.129016 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.129115 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.129221 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":56,"threads_started":1}
I20260809 03:34:08.129421 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.124266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.15.222.65:44339)
0809 03:34:08.124827 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.124838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.124863 (+ 25us) server_negotiation.cc:407] Connection header received
0809 03:34:08.124940 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.124945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.125156 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.125317 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:08.125776 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.126778 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.127971 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.128755 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.128862 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.129051 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.129160 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.129263 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":137,"threads_started":1}
I20260809 03:34:08.130268 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.124237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:44491 (local address 127.0.0.1:55556)
0809 03:34:08.125253 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.125265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.125338 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.126081 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.126084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.126102 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:08.126192 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.126198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.127506 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.127509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.128623 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.128632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.129902 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.129906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.129989 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.130011 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.130069 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.130134 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":53,"thread_start_us":102,"threads_started":1}
I20260809 03:34:08.130511 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.124839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.15.222.67:44491)
0809 03:34:08.125630 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.125641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.125664 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:34:08.125757 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.125762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.125951 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.126095 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:08.126296 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.127381 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.128732 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.129793 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.129893 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.130068 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.130113 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.130192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":162,"threads_started":1}
I20260809 03:34:08.130697 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.124752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42525 (local address 127.0.0.1:38654)
0809 03:34:08.125743 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.125755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.125854 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.126313 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.126315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.126328 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:08.126398 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.126403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.127661 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.127665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.129107 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.129118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.130314 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.130318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.130420 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.130442 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.130515 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.130579 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":325,"thread_start_us":97,"threads_started":1}
I20260809 03:34:08.130928 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.125186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.15.222.66:42525)
0809 03:34:08.125584 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.125599 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.125856 (+ 257us) server_negotiation.cc:407] Connection header received
0809 03:34:08.125960 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.125966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.126130 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.126255 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:08.126506 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.127565 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.129235 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.129991 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.130089 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.130250 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.130554 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.130652 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260809 03:34:08.132357 13158 tablet_service.cc:1467] Processing CreateTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 (DEFAULT_TABLE table=test-table [id=5557bfe418c74d86b25bb7d934720357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:08.133235 13158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1976aeac4d214e738e8dfc13a59f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:08.133253 13351 tablet_service.cc:1467] Processing CreateTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 (DEFAULT_TABLE table=test-table [id=5557bfe418c74d86b25bb7d934720357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:08.133253 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 (DEFAULT_TABLE table=test-table [id=5557bfe418c74d86b25bb7d934720357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:08.134276 13351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1976aeac4d214e738e8dfc13a59f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:08.134276 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1976aeac4d214e738e8dfc13a59f6b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:08.139297 13667 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Bootstrap starting.
I20260809 03:34:08.141057 13669 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Bootstrap starting.
I20260809 03:34:08.142024 13671 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Bootstrap starting.
I20260809 03:34:08.142179 13667 tablet_bootstrap.cc:654] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:08.143065 13667 log.cc:824] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:08.143149 13669 tablet_bootstrap.cc:654] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:08.143714 13671 tablet_bootstrap.cc:654] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:08.143865 13669 log.cc:824] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:08.144375 13671 log.cc:824] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:08.145172 13667 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: No bootstrap required, opened a new log
I20260809 03:34:08.145365 13667 ts_tablet_manager.cc:1397] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:34:08.146072 13669 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: No bootstrap required, opened a new log
I20260809 03:34:08.146250 13669 ts_tablet_manager.cc:1397] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 03:34:08.146562 13671 tablet_bootstrap.cc:492] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: No bootstrap required, opened a new log
I20260809 03:34:08.146739 13671 ts_tablet_manager.cc:1397] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:34:08.149348 13671 raft_consensus.cc:348] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.149560 13671 raft_consensus.cc:374] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:08.149613 13671 raft_consensus.cc:729] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c3c346df2c44d689ec8f10d9d4b19f, State: Initialized, Role: FOLLOWER
I20260809 03:34:08.149516 13667 raft_consensus.cc:348] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.149788 13667 raft_consensus.cc:374] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:08.149847 13667 raft_consensus.cc:729] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7de9c3bfe97408f8a314fdfa230fa9e, State: Initialized, Role: FOLLOWER
I20260809 03:34:08.150082 13671 consensus_queue.cc:260] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.150247 13667 consensus_queue.cc:260] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.150163 13669 raft_consensus.cc:348] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.150386 13669 raft_consensus.cc:374] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:08.150445 13669 raft_consensus.cc:729] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bd702471894a27a6d99deb7ecc19a2, State: Initialized, Role: FOLLOWER
I20260809 03:34:08.150830 13437 heartbeater.cc:503] Master 127.15.222.126:42431 was elected leader, sending a full tablet report...
I20260809 03:34:08.150862 13669 consensus_queue.cc:260] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.151017 13671 ts_tablet_manager.cc:1428] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 03:34:08.151554 13252 heartbeater.cc:503] Master 127.15.222.126:42431 was elected leader, sending a full tablet report...
I20260809 03:34:08.151641 13669 ts_tablet_manager.cc:1428] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:34:08.151641 13667 ts_tablet_manager.cc:1428] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260809 03:34:08.151955 13635 heartbeater.cc:503] Master 127.15.222.126:42431 was elected leader, sending a full tablet report...
W20260809 03:34:08.156418 13438 tablet.cc:2369] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 03:34:08.218425 13253 tablet.cc:2369] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:08.251693 13679 raft_consensus.cc:482] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:08.251886 13679 raft_consensus.cc:504] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.252985 13679 leader_election.cc:288] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bd702471894a27a6d99deb7ecc19a2 (127.15.222.67:44491), d7c3c346df2c44d689ec8f10d9d4b19f (127.15.222.66:42525)
I20260809 03:34:08.256613 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.253351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:54043 (local address 127.15.222.67:44491)
0809 03:34:08.253510 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.253515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.253621 (+ 106us) server_negotiation.cc:407] Connection header received
0809 03:34:08.253790 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.253793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.253834 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.253907 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:08.254208 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.254705 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.255743 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.256123 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.256227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.256280 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.256440 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.256503 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 03:34:08.256628 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.253204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:44491 (local address 127.15.222.65:54043)
0809 03:34:08.253535 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.253550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.253667 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.253943 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.253946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.253966 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:08.254093 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.254099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.254803 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.254806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.255611 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.255620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.256227 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.256231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.256320 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.256342 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.256411 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.256473 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260809 03:34:08.256791 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.253403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:51565 (local address 127.15.222.66:42525)
0809 03:34:08.253618 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.253622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.253860 (+ 238us) server_negotiation.cc:407] Connection header received
0809 03:34:08.253985 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.253988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.254030 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.254088 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:08.254337 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.254811 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.255776 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.256142 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.256241 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.256292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.256625 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.256691 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260809 03:34:08.256951 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.253206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42525 (local address 127.15.222.65:51565)
0809 03:34:08.253779 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.253791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:08.253874 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:08.254107 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:08.254110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:08.254125 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:08.254233 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.254239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.254914 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.254917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.255658 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:08.255666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.256241 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.256244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:08.256505 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.256524 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 03:34:08.256591 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 03:34:08.256655 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":27,"thread_start_us":57,"threads_started":1}
I20260809 03:34:08.257412 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1976aeac4d214e738e8dfc13a59f6b43" candidate_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bd702471894a27a6d99deb7ecc19a2" is_pre_election: true
I20260809 03:34:08.257499 13373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1976aeac4d214e738e8dfc13a59f6b43" candidate_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" is_pre_election: true
I20260809 03:34:08.257755 13565 raft_consensus.cc:2454] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7de9c3bfe97408f8a314fdfa230fa9e in term 0.
I20260809 03:34:08.257755 13373 raft_consensus.cc:2454] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7de9c3bfe97408f8a314fdfa230fa9e in term 0.
I20260809 03:34:08.258261 13091 leader_election.cc:302] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c3c346df2c44d689ec8f10d9d4b19f, e7de9c3bfe97408f8a314fdfa230fa9e; no voters:
I20260809 03:34:08.258570 13679 raft_consensus.cc:2781] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:08.258668 13679 raft_consensus.cc:482] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:08.258710 13679 raft_consensus.cc:3037] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:08.259838 13679 raft_consensus.cc:504] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.260177 13679 leader_election.cc:288] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [CANDIDATE]: Term 1 election: Requested vote from peers e7bd702471894a27a6d99deb7ecc19a2 (127.15.222.67:44491), d7c3c346df2c44d689ec8f10d9d4b19f (127.15.222.66:42525)
I20260809 03:34:08.260509 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1976aeac4d214e738e8dfc13a59f6b43" candidate_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bd702471894a27a6d99deb7ecc19a2"
I20260809 03:34:08.260510 13373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1976aeac4d214e738e8dfc13a59f6b43" candidate_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f"
I20260809 03:34:08.260640 13373 raft_consensus.cc:3037] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:08.260640 13565 raft_consensus.cc:3037] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:08.262028 13565 raft_consensus.cc:2454] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7de9c3bfe97408f8a314fdfa230fa9e in term 1.
I20260809 03:34:08.262028 13373 raft_consensus.cc:2454] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7de9c3bfe97408f8a314fdfa230fa9e in term 1.
I20260809 03:34:08.262346 13091 leader_election.cc:302] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d7c3c346df2c44d689ec8f10d9d4b19f, e7bd702471894a27a6d99deb7ecc19a2, e7de9c3bfe97408f8a314fdfa230fa9e; no voters:
I20260809 03:34:08.262547 13679 raft_consensus.cc:2781] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:08.262763 13679 raft_consensus.cc:686] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 1 LEADER]: Becoming Leader. State: Replica: e7de9c3bfe97408f8a314fdfa230fa9e, State: Running, Role: LEADER
I20260809 03:34:08.263087 13679 consensus_queue.cc:237] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } }
I20260809 03:34:08.266162 12961 catalog_manager.cc:5495] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e reported cstate change: term changed from 0 to 1, leader changed from <none> to e7de9c3bfe97408f8a314fdfa230fa9e (127.15.222.65). New cstate: current_term: 1 leader_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7de9c3bfe97408f8a314fdfa230fa9e" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 44339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 } health_report { overall_health: UNKNOWN } } }
I20260809 03:34:08.279780 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.276406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.15.222.65:44339)
0809 03:34:08.276522 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:08.276526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:08.276540 (+ 14us) server_negotiation.cc:407] Connection header received
0809 03:34:08.276676 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:08.276679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:08.276730 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:08.276816 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:08.277049 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.277547 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.278416 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:08.278634 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:08.278720 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:08.279008 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:08.279395 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:08.279525 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:08.279625 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 03:34:08.279658 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:34:08.285473 13373 raft_consensus.cc:1261] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 1 FOLLOWER]: Refusing update from remote peer e7de9c3bfe97408f8a314fdfa230fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:08.285475 13565 raft_consensus.cc:1261] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 1 FOLLOWER]: Refusing update from remote peer e7de9c3bfe97408f8a314fdfa230fa9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:08.286258 13679 consensus_queue.cc:1035] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c3c346df2c44d689ec8f10d9d4b19f" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:08.286468 13686 consensus_queue.cc:1035] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bd702471894a27a6d99deb7ecc19a2" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:08.292912 13689 mvcc.cc:205] Tried to move back new op lower bound from 7316465452169666560 to 7316465452088102912. Current Snapshot: MvccSnapshot[applied={T|T < 7316465452169666560}]
I20260809 03:34:08.293372 13692 mvcc.cc:205] Tried to move back new op lower bound from 7316465452169666560 to 7316465452088102912. Current Snapshot: MvccSnapshot[applied={T|T < 7316465452169666560}]
I20260809 03:34:08.295639 13694 mvcc.cc:205] Tried to move back new op lower bound from 7316465452169666560 to 7316465452088102912. Current Snapshot: MvccSnapshot[applied={T|T < 7316465452169666560}]
I20260809 03:34:08.301391 12961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:08.301527 12961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:08.303735 12961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cfc4fdb70c847cf987263ced713df81: Sending DeleteTablet for 3 replicas of tablet 1976aeac4d214e738e8dfc13a59f6b43
I20260809 03:34:08.304441 13158 tablet_service.cc:1514] Processing DeleteTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:08 UTC) from {username='slave'} at 127.0.0.1:59170
I20260809 03:34:08.304481 13351 tablet_service.cc:1514] Processing DeleteTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:08 UTC) from {username='slave'} at 127.0.0.1:38654
I20260809 03:34:08.304934 13707 tablet_replica.cc:330] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e: stopping tablet replica
I20260809 03:34:08.304900 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1976aeac4d214e738e8dfc13a59f6b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:08 UTC) from {username='slave'} at 127.0.0.1:55556
I20260809 03:34:08.305227 13707 raft_consensus.cc:2229] T 1976aeac4d214e738e8dfc13a59f6b43 P e7de9c3bfe97408f8a314fdfa230fa9e [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:34:08.305290 13710 tablet_replica.cc:330] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: stopping tablet replica
I20260809 03:34:08.305521 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 13059
I20260809 03:34:08.305558 13710 raft_consensus.cc:2229] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:08.309058 13710 raft_consensus.cc:2258] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:08.309762 13709 tablet_replica.cc:330] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: stopping tablet replica
I20260809 03:34:08.310038 13709 raft_consensus.cc:2229] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:08.310307 13709 raft_consensus.cc:2258] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:08.310384 13710 ts_tablet_manager.cc:1905] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:08.311519 13709 ts_tablet_manager.cc:1905] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:08.312708 13710 ts_tablet_manager.cc:1918] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:08.312793 13710 log.cc:1196] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/1976aeac4d214e738e8dfc13a59f6b43
I20260809 03:34:08.313043 13710 ts_tablet_manager.cc:1939] T 1976aeac4d214e738e8dfc13a59f6b43 P e7bd702471894a27a6d99deb7ecc19a2: Deleting consensus metadata
I20260809 03:34:08.314061 13709 ts_tablet_manager.cc:1918] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:08.314045 12945 catalog_manager.cc:4841] TS e7bd702471894a27a6d99deb7ecc19a2 (127.15.222.67:44491): tablet 1976aeac4d214e738e8dfc13a59f6b43 (table test-table [id=5557bfe418c74d86b25bb7d934720357]) successfully deleted
I20260809 03:34:08.314152 13709 log.cc:1196] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/1976aeac4d214e738e8dfc13a59f6b43
I20260809 03:34:08.314404 13709 ts_tablet_manager.cc:1939] T 1976aeac4d214e738e8dfc13a59f6b43 P d7c3c346df2c44d689ec8f10d9d4b19f: Deleting consensus metadata
I20260809 03:34:08.315413 12946 catalog_manager.cc:4841] TS d7c3c346df2c44d689ec8f10d9d4b19f (127.15.222.66:42525): tablet 1976aeac4d214e738e8dfc13a59f6b43 (table test-table [id=5557bfe418c74d86b25bb7d934720357]) successfully deleted
W20260809 03:34:08.316366 12944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:44339 (error 108)
I20260809 03:34:08.316460 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 13255
I20260809 03:34:08.323551 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:08.323294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:44339 (local address 127.0.0.1:59190)
0809 03:34:08.323413 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:08.323481 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 03:34:08.323760 12944 catalog_manager.cc:4584] TS e7de9c3bfe97408f8a314fdfa230fa9e (127.15.222.65:44339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1976aeac4d214e738e8dfc13a59f6b43: Network error: Client connection negotiation failed: client connection to 127.15.222.65:44339: connect: Connection refused (error 111)
I20260809 03:34:08.332407 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 13440
I20260809 03:34:08.345609 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 12918
2026-08-09T03:34:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1063 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:08 dist-test-slave-cvwc krb5kdc[13716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:08 dist-test-slave-cvwc krb5kdc[13716](info): set up 2 sockets
Aug 09 03:34:08 dist-test-slave-cvwc krb5kdc[13716](info): commencing operation
krb5kdc: starting...
W20260809 03:34:09.476260 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:09 dist-test-slave-cvwc krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:09.590521 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37857
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:09.700776 13738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:09.701028 13738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:09.701067 13738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:09.705031 13738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:09.705089 13738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:09.705107 13738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:09.705122 13738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:09.705137 13738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:09.710296 13738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:09.711259 13738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:09.712280 13738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:09.720501 13745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:09.720499 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:09.720587 13738 server_base.cc:1034] running on GCE node
W20260809 03:34:09.720487 13746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:09.721302 13738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:09.722376 13738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:09.723532 13738 hybrid_clock.cc:648] HybridClock initialized: now 1786246449723513 us; error 28 us; skew 500 ppm
Aug 09 03:34:09 dist-test-slave-cvwc krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246449, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:09.726189 13738 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:09.727021 13738 webserver.cc:460] Webserver started at http://127.15.222.126:37107/ using document root <none> and password file <none>
I20260809 03:34:09.727542 13738 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:09.727608 13738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:09.727816 13738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:09.729862 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "446d189be26344268a941d7b2bdb4638"
format_stamp: "Formatted at 2026-08-09 03:34:09 on dist-test-slave-cvwc"
server_key: "8c66b70e570ac3c018f53bc0102c8fce"
server_key_iv: "045705f525712d0191c009446e0e145a"
server_key_version: "encryptionkey@0"
I20260809 03:34:09.730410 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "446d189be26344268a941d7b2bdb4638"
format_stamp: "Formatted at 2026-08-09 03:34:09 on dist-test-slave-cvwc"
server_key: "8c66b70e570ac3c018f53bc0102c8fce"
server_key_iv: "045705f525712d0191c009446e0e145a"
server_key_version: "encryptionkey@0"
I20260809 03:34:09.734037 13738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 03:34:09.736302 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:09.737468 13738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:09.737582 13738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "446d189be26344268a941d7b2bdb4638"
format_stamp: "Formatted at 2026-08-09 03:34:09 on dist-test-slave-cvwc"
server_key: "8c66b70e570ac3c018f53bc0102c8fce"
server_key_iv: "045705f525712d0191c009446e0e145a"
server_key_version: "encryptionkey@0"
I20260809 03:34:09.737679 13738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:09.764912 13738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:09.768049 13738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:09.768229 13738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:09.774981 13738 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:37857
I20260809 03:34:09.774995 13843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:37857 every 8 connection(s)
I20260809 03:34:09.775907 13738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:09.776343 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 13738
I20260809 03:34:09.776468 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:09.776741 16249 external_mini_cluster.cc:1428] Setting key a64c9d247d20e9ea32df11ea3a06a5e4
I20260809 03:34:09.779078 13844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:09.785845 13844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: Bootstrap starting.
Aug 09 03:34:09 dist-test-slave-cvwc krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:09.788004 13844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:09.788695 13844 log.cc:824] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:09.790642 13844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: No bootstrap required, opened a new log
I20260809 03:34:09.790836 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:09.778219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.15.222.126:37857)
0809 03:34:09.778688 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:09.778703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:09.778740 (+ 37us) server_negotiation.cc:407] Connection header received
0809 03:34:09.779414 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:09.779432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:09.779757 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:09.780042 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:09.781078 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:09.782147 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:09.783281 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:09.783745 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:09.783862 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:09.786638 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:09.786643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:09.786655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:09.786682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:09.788237 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:09.788635 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:09.788640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:09.788645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:09.788753 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:09.789006 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:09.789009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:09.789011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:09.789388 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:09.789570 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:09.789864 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 03:34:09.790142 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260809 03:34:09.793515 13844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } }
I20260809 03:34:09.793746 13844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:09.793819 13844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d189be26344268a941d7b2bdb4638, State: Initialized, Role: FOLLOWER
I20260809 03:34:09.794181 13844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } }
I20260809 03:34:09.794292 13844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:09.794334 13844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:09.794400 13844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:09.795164 13844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } }
I20260809 03:34:09.795461 13844 leader_election.cc:302] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 446d189be26344268a941d7b2bdb4638; no voters:
I20260809 03:34:09.795699 13844 leader_election.cc:288] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:09.795792 13855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:09.796006 13855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [term 1 LEADER]: Becoming Leader. State: Replica: 446d189be26344268a941d7b2bdb4638, State: Running, Role: LEADER
I20260809 03:34:09.796289 13855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } }
I20260809 03:34:09.796826 13844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:09.797926 13858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446d189be26344268a941d7b2bdb4638. Latest consensus state: current_term: 1 leader_uuid: "446d189be26344268a941d7b2bdb4638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } } }
I20260809 03:34:09.798063 13858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:09.798353 13856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446d189be26344268a941d7b2bdb4638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d189be26344268a941d7b2bdb4638" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37857 } } }
I20260809 03:34:09.798450 13856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:09.798645 13868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:09.801137 13868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:09.805871 13868 catalog_manager.cc:1347] Generated new cluster ID: 31780b1c0d0648cdbc023d439e9fc8db
I20260809 03:34:09.805941 13868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:09.814072 13868 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:09.814891 13868 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:09.821022 13868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: Generated new TSK 0
I20260809 03:34:09.821658 13868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
Entry for principal kudu/127.15.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab.
I20260809 03:34:09.860054 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:10.003125 13892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:10.003381 13892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:10.003422 13892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:10.007319 13892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:10.007375 13892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:10.007444 13892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:10.012631 13892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.65.keytab
--principal=kudu/127.15.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:10.013654 13892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:10.014688 13892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:10.024094 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.024094 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.024129 13899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:10.024171 13892 server_base.cc:1034] running on GCE node
I20260809 03:34:10.024752 13892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:10.025357 13892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:10.026510 13892 hybrid_clock.cc:648] HybridClock initialized: now 1786246450026491 us; error 28 us; skew 500 ppm
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:10.029134 13892 init.cc:377] Logged in from keytab as kudu/127.15.222.65@KRBTEST.COM (short username kudu)
I20260809 03:34:10.029965 13892 webserver.cc:460] Webserver started at http://127.15.222.65:35589/ using document root <none> and password file <none>
I20260809 03:34:10.030470 13892 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:10.030526 13892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:10.030690 13892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:10.032593 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "40d01a9631ed4353bb6955778c6fd7f8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "70ce90156f04df5d9c64ce42fcc41f62"
server_key_iv: "23df3ea7bc7e5d4aa7cd362bc3f49e38"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.033087 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "40d01a9631ed4353bb6955778c6fd7f8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "70ce90156f04df5d9c64ce42fcc41f62"
server_key_iv: "23df3ea7bc7e5d4aa7cd362bc3f49e38"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.036490 13892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:10.038666 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.039749 13892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:10.039852 13892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "40d01a9631ed4353bb6955778c6fd7f8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "70ce90156f04df5d9c64ce42fcc41f62"
server_key_iv: "23df3ea7bc7e5d4aa7cd362bc3f49e38"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.039945 13892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:10.055091 13892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:10.058121 13892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:10.058296 13892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:10.058866 13892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:10.059666 13892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:10.059712 13892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.059763 13892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:10.059791 13892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.067762 13892 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:39425
I20260809 03:34:10.067790 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:39425 every 8 connection(s)
I20260809 03:34:10.068601 13892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:10.076871 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 13892
I20260809 03:34:10.076996 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:34:10.077262 16249 external_mini_cluster.cc:1428] Setting key 5ae4ba3f452ef577b64ee468d6ee3548
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.65@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:10.081888 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.070333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:32839 (local address 127.15.222.126:37857)
0809 03:34:10.070478 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.070482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.071330 (+ 848us) server_negotiation.cc:407] Connection header received
0809 03:34:10.072616 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.072618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.072665 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.072736 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:10.073863 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.074392 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.075693 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.075933 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.076006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.078433 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:10.078435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:10.078437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:10.078461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:10.079941 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.080388 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.080392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.080394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.080484 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.080783 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.080786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.080788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.080990 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:10.081078 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.081570 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.081700 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:34:10.082691 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.070728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37857 (local address 127.15.222.65:32839)
0809 03:34:10.071187 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.071231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.072429 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.072882 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.072888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.073227 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:10.073696 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.073742 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.074509 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.074512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.075592 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.075602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.076044 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.076046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.076183 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.076694 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:10.076709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:10.078279 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:10.080045 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.080050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.080061 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.080302 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.080563 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.080565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.080566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.080691 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.081090 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:10.081097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:10.081332 (+ 235us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.081507 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.081861 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260809 03:34:10.083853 14094 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37857
I20260809 03:34:10.084082 14094 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:10.084542 14094 heartbeater.cc:511] Master 127.15.222.126:37857 requested a full tablet report, sending...
I20260809 03:34:10.086019 13788 ts_manager.cc:194] Registered new tserver with Master: 40d01a9631ed4353bb6955778c6fd7f8 (127.15.222.65:39425)
I20260809 03:34:10.087111 13788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.65@KRBTEST.COM'} at 127.15.222.65:32839
WARNING: no policy specified for kudu/127.15.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
Entry for principal kudu/127.15.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab.
I20260809 03:34:10.102344 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:10.209496 14100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:10.209764 14100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:10.209829 14100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:10.213703 14100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:10.213774 14100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:10.213850 14100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:10.218883 14100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.66.keytab
--principal=kudu/127.15.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:10.219805 14100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:10.220741 14100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:10.229859 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.229831 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.229832 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:10.230296 14100 server_base.cc:1034] running on GCE node
I20260809 03:34:10.230695 14100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:10.231304 14100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:10.232460 14100 hybrid_clock.cc:648] HybridClock initialized: now 1786246450232439 us; error 32 us; skew 500 ppm
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:10.235127 14100 init.cc:377] Logged in from keytab as kudu/127.15.222.66@KRBTEST.COM (short username kudu)
I20260809 03:34:10.235968 14100 webserver.cc:460] Webserver started at http://127.15.222.66:43925/ using document root <none> and password file <none>
I20260809 03:34:10.236496 14100 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:10.236562 14100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:10.236758 14100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:10.238832 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "3660cec051a742b1b207f68e10839f5d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "6fb967404cda28f6cd7c6810a5a8fb9f"
server_key_iv: "b10f162cccc408e4e1b0641cc01b2518"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.239377 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "3660cec051a742b1b207f68e10839f5d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "6fb967404cda28f6cd7c6810a5a8fb9f"
server_key_iv: "b10f162cccc408e4e1b0641cc01b2518"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.242992 14100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:10.245270 14126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.246402 14100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:10.246512 14100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "3660cec051a742b1b207f68e10839f5d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "6fb967404cda28f6cd7c6810a5a8fb9f"
server_key_iv: "b10f162cccc408e4e1b0641cc01b2518"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.246598 14100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:10.274806 14100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:10.277922 14100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:10.278086 14100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:10.278666 14100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:10.279485 14100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:10.279531 14100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.279584 14100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:10.279609 14100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.287791 14100 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:33617
I20260809 03:34:10.287844 14291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:33617 every 8 connection(s)
I20260809 03:34:10.288631 14100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.66@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:10.297991 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 14100
I20260809 03:34:10.298120 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:10.298456 16249 external_mini_cluster.cc:1428] Setting key 45934d6a66f002dce756423a8f82d1b5
I20260809 03:34:10.301662 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.290300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:43781 (local address 127.15.222.126:37857)
0809 03:34:10.290426 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.290430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.291297 (+ 867us) server_negotiation.cc:407] Connection header received
0809 03:34:10.292570 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.292573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.292630 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.292702 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:10.293812 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.294337 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.295429 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.295649 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.295734 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.298111 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:10.298115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:10.298118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:10.298146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:10.299707 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.300247 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.300249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.300251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.300325 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.300654 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.300656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.300657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.300811 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:10.300877 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.301406 (+ 529us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.301511 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 03:34:10.302541 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.290721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37857 (local address 127.15.222.66:43781)
0809 03:34:10.291153 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.291199 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.292376 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.292876 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.292883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.293201 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:10.293655 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.293668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.294455 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.294458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.295308 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.295318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.295779 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.295783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.295951 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.296420 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:10.296436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:10.297938 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:10.299818 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.299823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.299834 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.300159 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.300408 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.300411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.300413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.300573 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.300901 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:10.300908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:10.301188 (+ 280us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.301369 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.301668 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260809 03:34:10.303640 14295 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37857
I20260809 03:34:10.303850 14295 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:10.304286 14295 heartbeater.cc:511] Master 127.15.222.126:37857 requested a full tablet report, sending...
I20260809 03:34:10.305318 13788 ts_manager.cc:194] Registered new tserver with Master: 3660cec051a742b1b207f68e10839f5d (127.15.222.66:33617)
I20260809 03:34:10.306089 13788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.66@KRBTEST.COM'} at 127.15.222.66:43781
WARNING: no policy specified for kudu/127.15.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
Entry for principal kudu/127.15.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab.
I20260809 03:34:10.324219 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:10.435451 14301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:10.435688 14301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:10.435737 14301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:10.439661 14301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:10.439714 14301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:10.439798 14301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:10.445004 14301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.67.keytab
--principal=kudu/127.15.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:10.446009 14301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:10.447003 14301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:10.456400 14312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.456404 14310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.456409 14309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:10.456590 14301 server_base.cc:1034] running on GCE node
I20260809 03:34:10.457070 14301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:10.457607 14301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:10.458753 14301 hybrid_clock.cc:648] HybridClock initialized: now 1786246450458726 us; error 38 us; skew 500 ppm
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:10.461299 14301 init.cc:377] Logged in from keytab as kudu/127.15.222.67@KRBTEST.COM (short username kudu)
I20260809 03:34:10.462148 14301 webserver.cc:460] Webserver started at http://127.15.222.67:39571/ using document root <none> and password file <none>
I20260809 03:34:10.462678 14301 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:10.462742 14301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:10.462925 14301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:10.464813 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "48a83c1507f5482bbb9bc61ac156910d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "beb69f939bfd2057f1f0d53fc119ac08"
server_key_iv: "87930499235b6403ad6b3db684955aea"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.465343 14301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "48a83c1507f5482bbb9bc61ac156910d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "beb69f939bfd2057f1f0d53fc119ac08"
server_key_iv: "87930499235b6403ad6b3db684955aea"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.468734 14301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:10.471030 14323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.472231 14301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:10.472354 14301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "48a83c1507f5482bbb9bc61ac156910d"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "beb69f939bfd2057f1f0d53fc119ac08"
server_key_iv: "87930499235b6403ad6b3db684955aea"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.472443 14301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:10.490883 14301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:10.493804 14301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:10.493961 14301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:10.494486 14301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:10.495246 14301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:10.495296 14301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.495347 14301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:10.495371 14301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.503888 14301 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:43025
I20260809 03:34:10.503921 14509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:43025 every 8 connection(s)
I20260809 03:34:10.504679 14301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:10.509832 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 14301
I20260809 03:34:10.509950 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:10.510221 16249 external_mini_cluster.cc:1428] Setting key 949cb5b9b1d70a7ddbdaff15eb338622
Aug 09 03:34:10 dist-test-slave-cvwc krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.67@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:10.516961 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.506314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:40537 (local address 127.15.222.126:37857)
0809 03:34:10.506441 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.506445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.507582 (+ 1137us) server_negotiation.cc:407] Connection header received
0809 03:34:10.508421 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.508424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.508483 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.508552 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:10.509615 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.510258 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.511523 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.511743 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.511827 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.514073 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:10.514077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:10.514080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:10.514104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:10.515129 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.515572 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.515576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.515577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.515653 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.515975 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.515978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.515980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.516151 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:10.516235 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.516675 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.516792 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:10.517632 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.506623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:37857 (local address 127.15.222.67:40537)
0809 03:34:10.507453 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.507491 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.508227 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.508715 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.508722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.509045 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:10.509478 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.509491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.510367 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.510371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.511418 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.511427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.511866 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.511870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.512059 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.512506 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:10.512522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:10.513933 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:10.515228 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.515233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.515247 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.515484 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.515731 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:10.515734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:10.515737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:10.515877 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:10.516246 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:10.516252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:10.516465 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.516616 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.516917 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":535,"threads_started":1}
I20260809 03:34:10.518595 14510 heartbeater.cc:347] Connected to a master server at 127.15.222.126:37857
I20260809 03:34:10.518808 14510 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:10.519317 14510 heartbeater.cc:511] Master 127.15.222.126:37857 requested a full tablet report, sending...
I20260809 03:34:10.520287 13788 ts_manager.cc:194] Registered new tserver with Master: 48a83c1507f5482bbb9bc61ac156910d (127.15.222.67:43025)
I20260809 03:34:10.520817 13788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.222.67@KRBTEST.COM'} at 127.15.222.67:40537
I20260809 03:34:10.523485 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:10.531337 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.525231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.15.222.126:37857)
0809 03:34:10.525343 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.525347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.525521 (+ 174us) server_negotiation.cc:407] Connection header received
0809 03:34:10.525606 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.525609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.525643 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.525701 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:10.526317 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.526793 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.527515 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.527701 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.527762 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.528875 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:10.528878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:10.528880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:10.528903 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:10.529996 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.530338 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.530340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.530341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.530415 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:10.530682 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:10.530683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:10.530684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:10.530867 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:10.530962 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.531126 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.531211 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:34:10.533794 13788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:10.536599 13788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:10.549752 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.544451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43025 (local address 127.0.0.1:57124)
0809 03:34:10.545071 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.545119 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.545243 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.546088 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.546094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.546115 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:10.546230 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.546236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.547296 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.547299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.548381 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.548398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.549161 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.549164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.549310 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.549353 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.549462 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.549584 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":67,"threads_started":1}
I20260809 03:34:10.549757 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.545074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.15.222.67:43025)
0809 03:34:10.545641 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.545654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.545676 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:34:10.545776 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.545781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.545947 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.546086 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:10.546331 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.547182 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.548497 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.549049 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.549145 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.549353 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.549502 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.549595 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":147,"threads_started":1}
I20260809 03:34:10.551316 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.544580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:33617 (local address 127.0.0.1:40068)
0809 03:34:10.545501 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.545515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.545594 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.546521 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.546524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.546543 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:10.546654 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.546661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.548128 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.548131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.549494 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.549506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.550914 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.550918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.551016 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.551039 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.551104 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.551175 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":75,"thread_start_us":102,"threads_started":1}
I20260809 03:34:10.551700 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.545183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.15.222.65:39425)
0809 03:34:10.546216 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.546230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.546251 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:34:10.546321 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.546326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.546490 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.546637 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:10.546851 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.547977 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.549632 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.550764 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.550876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.551010 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.551382 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.551567 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":85,"threads_started":1}
I20260809 03:34:10.551961 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.544793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39425 (local address 127.0.0.1:58566)
0809 03:34:10.545872 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.545900 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.545988 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.546624 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.546627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.546647 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:10.546746 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.546751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.548128 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.548131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.549516 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.549524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.551065 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.551068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.551147 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.551167 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.551223 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.551277 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":98,"thread_start_us":114,"threads_started":1}
I20260809 03:34:10.551965 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.545472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.15.222.66:33617)
0809 03:34:10.545912 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.545926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.545948 (+ 22us) server_negotiation.cc:407] Connection header received
0809 03:34:10.546020 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.546025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.546210 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.546351 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:10.546775 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.547977 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.549632 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.550607 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.550709 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.550851 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.551662 (+ 811us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.551837 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":99,"threads_started":1}
I20260809 03:34:10.552659 14403 tablet_service.cc:1467] Processing CreateTablet for tablet f387cc705cdd483abaaca5546591c2e0 (DEFAULT_TABLE table=test-table [id=754dcf21be6844549d2252a95ee8ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:10.553831 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387cc705cdd483abaaca5546591c2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:10.554853 13990 tablet_service.cc:1467] Processing CreateTablet for tablet f387cc705cdd483abaaca5546591c2e0 (DEFAULT_TABLE table=test-table [id=754dcf21be6844549d2252a95ee8ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:10.555872 13990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387cc705cdd483abaaca5546591c2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:10.559297 14201 tablet_service.cc:1467] Processing CreateTablet for tablet f387cc705cdd483abaaca5546591c2e0 (DEFAULT_TABLE table=test-table [id=754dcf21be6844549d2252a95ee8ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:10.560249 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387cc705cdd483abaaca5546591c2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:10.561198 14537 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Bootstrap starting.
I20260809 03:34:10.562673 14538 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: Bootstrap starting.
I20260809 03:34:10.563663 14537 tablet_bootstrap.cc:654] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:10.564426 14538 tablet_bootstrap.cc:654] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:10.565001 14537 log.cc:824] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:10.565614 14538 log.cc:824] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:10.567114 14537 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: No bootstrap required, opened a new log
I20260809 03:34:10.567097 14541 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Bootstrap starting.
I20260809 03:34:10.567305 14537 ts_tablet_manager.cc:1397] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 03:34:10.567423 14538 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: No bootstrap required, opened a new log
I20260809 03:34:10.567605 14538 ts_tablet_manager.cc:1397] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 03:34:10.569387 14541 tablet_bootstrap.cc:654] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:10.570281 14537 raft_consensus.cc:348] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.570492 14537 raft_consensus.cc:374] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:10.570552 14537 raft_consensus.cc:729] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a83c1507f5482bbb9bc61ac156910d, State: Initialized, Role: FOLLOWER
I20260809 03:34:10.570623 14541 log.cc:824] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:10.570993 14537 consensus_queue.cc:260] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.571291 14538 raft_consensus.cc:348] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.571552 14538 raft_consensus.cc:374] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:10.571611 14538 raft_consensus.cc:729] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d01a9631ed4353bb6955778c6fd7f8, State: Initialized, Role: FOLLOWER
I20260809 03:34:10.571712 14510 heartbeater.cc:503] Master 127.15.222.126:37857 was elected leader, sending a full tablet report...
I20260809 03:34:10.571976 14537 ts_tablet_manager.cc:1428] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:34:10.572008 14538 consensus_queue.cc:260] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.572942 14538 ts_tablet_manager.cc:1428] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260809 03:34:10.573109 14541 tablet_bootstrap.cc:492] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: No bootstrap required, opened a new log
I20260809 03:34:10.573143 14094 heartbeater.cc:503] Master 127.15.222.126:37857 was elected leader, sending a full tablet report...
I20260809 03:34:10.573274 14541 ts_tablet_manager.cc:1397] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260809 03:34:10.576608 14541 raft_consensus.cc:348] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.576799 14541 raft_consensus.cc:374] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:10.576860 14541 raft_consensus.cc:729] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3660cec051a742b1b207f68e10839f5d, State: Initialized, Role: FOLLOWER
I20260809 03:34:10.577210 14541 consensus_queue.cc:260] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.577776 14295 heartbeater.cc:503] Master 127.15.222.126:37857 was elected leader, sending a full tablet report...
I20260809 03:34:10.577943 14541 ts_tablet_manager.cc:1428] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260809 03:34:10.674664 14545 raft_consensus.cc:482] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:10.674870 14545 raft_consensus.cc:504] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.676007 14545 leader_election.cc:288] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48a83c1507f5482bbb9bc61ac156910d (127.15.222.67:43025), 40d01a9631ed4353bb6955778c6fd7f8 (127.15.222.65:39425)
I20260809 03:34:10.679827 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.676401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:40087 (local address 127.15.222.65:39425)
0809 03:34:10.676543 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.676547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.676880 (+ 333us) server_negotiation.cc:407] Connection header received
0809 03:34:10.677003 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.677006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.677063 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.677121 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:10.677371 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.677942 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.678951 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.679343 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.679460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.679510 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.679652 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.679724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 03:34:10.679833 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.676265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:43025 (local address 127.15.222.66:41701)
0809 03:34:10.676400 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.676416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.676517 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.676791 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.676793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.676813 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:10.676944 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.676951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.677789 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.677793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.678680 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.678690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.679355 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.679359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.679464 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.679488 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.679558 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.679625 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260809 03:34:10.679987 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.676275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39425 (local address 127.15.222.66:40087)
0809 03:34:10.676789 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.676803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:10.676910 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:10.677144 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:10.677147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:10.677164 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:10.677280 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.677286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.678037 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.678040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.678828 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:10.678838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.679457 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.679460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:10.679531 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.679555 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:34:10.679630 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 03:34:10.679689 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":175,"thread_start_us":149,"threads_started":1}
I20260809 03:34:10.679802 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.676307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:41701 (local address 127.15.222.67:43025)
0809 03:34:10.676446 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.676451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.676469 (+ 18us) server_negotiation.cc:407] Connection header received
0809 03:34:10.676627 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.676630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.676688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.676772 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:10.677053 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.677661 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.678804 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.679242 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.679335 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.679377 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.679586 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.679668 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 03:34:10.680634 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387cc705cdd483abaaca5546591c2e0" candidate_uuid: "3660cec051a742b1b207f68e10839f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a83c1507f5482bbb9bc61ac156910d" is_pre_election: true
I20260809 03:34:10.680634 14022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387cc705cdd483abaaca5546591c2e0" candidate_uuid: "3660cec051a742b1b207f68e10839f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d01a9631ed4353bb6955778c6fd7f8" is_pre_election: true
I20260809 03:34:10.680917 14022 raft_consensus.cc:2454] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3660cec051a742b1b207f68e10839f5d in term 0.
I20260809 03:34:10.680922 14435 raft_consensus.cc:2454] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3660cec051a742b1b207f68e10839f5d in term 0.
I20260809 03:34:10.681399 14129 leader_election.cc:302] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3660cec051a742b1b207f68e10839f5d, 40d01a9631ed4353bb6955778c6fd7f8; no voters:
I20260809 03:34:10.681660 14545 raft_consensus.cc:2781] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:10.681773 14545 raft_consensus.cc:482] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:10.681825 14545 raft_consensus.cc:3037] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:10.682837 14545 raft_consensus.cc:504] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.683151 14545 leader_election.cc:288] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [CANDIDATE]: Term 1 election: Requested vote from peers 48a83c1507f5482bbb9bc61ac156910d (127.15.222.67:43025), 40d01a9631ed4353bb6955778c6fd7f8 (127.15.222.65:39425)
I20260809 03:34:10.683477 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387cc705cdd483abaaca5546591c2e0" candidate_uuid: "3660cec051a742b1b207f68e10839f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a83c1507f5482bbb9bc61ac156910d"
I20260809 03:34:10.683477 14022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387cc705cdd483abaaca5546591c2e0" candidate_uuid: "3660cec051a742b1b207f68e10839f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d01a9631ed4353bb6955778c6fd7f8"
I20260809 03:34:10.683598 14435 raft_consensus.cc:3037] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:10.683598 14022 raft_consensus.cc:3037] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:10.684831 14435 raft_consensus.cc:2454] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3660cec051a742b1b207f68e10839f5d in term 1.
I20260809 03:34:10.684831 14022 raft_consensus.cc:2454] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3660cec051a742b1b207f68e10839f5d in term 1.
I20260809 03:34:10.685165 14130 leader_election.cc:302] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3660cec051a742b1b207f68e10839f5d, 48a83c1507f5482bbb9bc61ac156910d; no voters:
I20260809 03:34:10.685341 14545 raft_consensus.cc:2781] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:10.685564 14545 raft_consensus.cc:686] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 1 LEADER]: Becoming Leader. State: Replica: 3660cec051a742b1b207f68e10839f5d, State: Running, Role: LEADER
I20260809 03:34:10.685961 14545 consensus_queue.cc:237] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } }
I20260809 03:34:10.688971 13786 catalog_manager.cc:5495] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3660cec051a742b1b207f68e10839f5d (127.15.222.66). New cstate: current_term: 1 leader_uuid: "3660cec051a742b1b207f68e10839f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3660cec051a742b1b207f68e10839f5d" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 33617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 } health_report { overall_health: UNKNOWN } } }
I20260809 03:34:10.699258 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.695728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.15.222.66:33617)
0809 03:34:10.695847 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.695851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.695863 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:10.695989 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.695992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.696044 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.696128 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:10.696380 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.696933 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.697822 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.698049 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.698135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.698433 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:10.698887 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:10.699000 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.699062 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.699115 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:10.704777 14435 raft_consensus.cc:1261] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 1 FOLLOWER]: Refusing update from remote peer 3660cec051a742b1b207f68e10839f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:10.704778 14022 raft_consensus.cc:1261] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 1 FOLLOWER]: Refusing update from remote peer 3660cec051a742b1b207f68e10839f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:10.705437 14550 consensus_queue.cc:1035] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d01a9631ed4353bb6955778c6fd7f8" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:10.705598 14545 consensus_queue.cc:1035] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a83c1507f5482bbb9bc61ac156910d" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:10.712601 14556 mvcc.cc:205] Tried to move back new op lower bound from 7316465462080151552 to 7316465462012313600. Current Snapshot: MvccSnapshot[applied={T|T < 7316465462080151552}]
I20260809 03:34:10.719815 13788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:10.719956 13788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:10.722113 13788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 446d189be26344268a941d7b2bdb4638: Sending DeleteTablet for 3 replicas of tablet f387cc705cdd483abaaca5546591c2e0
I20260809 03:34:10.722800 14403 tablet_service.cc:1514] Processing DeleteTablet for tablet f387cc705cdd483abaaca5546591c2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:10 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:57124
I20260809 03:34:10.722800 14201 tablet_service.cc:1514] Processing DeleteTablet for tablet f387cc705cdd483abaaca5546591c2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:10 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:40068
I20260809 03:34:10.723274 14572 tablet_replica.cc:330] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: stopping tablet replica
I20260809 03:34:10.723446 13990 tablet_service.cc:1514] Processing DeleteTablet for tablet f387cc705cdd483abaaca5546591c2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:10 UTC) from {username='kudu', principal='kudu/127.15.222.126@KRBTEST.COM'} at 127.0.0.1:58566
I20260809 03:34:10.723690 14572 raft_consensus.cc:2229] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:34:10.723852 14574 tablet_replica.cc:330] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8: stopping tablet replica
I20260809 03:34:10.723980 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 13892
I20260809 03:34:10.724128 14574 raft_consensus.cc:2229] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:10.724380 14574 raft_consensus.cc:2258] T f387cc705cdd483abaaca5546591c2e0 P 40d01a9631ed4353bb6955778c6fd7f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:10.731030 14572 raft_consensus.cc:2258] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:10.731458 14570 tablet_replica.cc:330] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: stopping tablet replica
I20260809 03:34:10.731752 14570 raft_consensus.cc:2229] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:10.732043 14570 raft_consensus.cc:2258] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:10.734807 14570 ts_tablet_manager.cc:1905] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:10.737537 14570 ts_tablet_manager.cc:1918] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:10.737684 14570 log.cc:1196] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/f387cc705cdd483abaaca5546591c2e0
I20260809 03:34:10.738109 14570 ts_tablet_manager.cc:1939] T f387cc705cdd483abaaca5546591c2e0 P 48a83c1507f5482bbb9bc61ac156910d: Deleting consensus metadata
I20260809 03:34:10.739421 13764 catalog_manager.cc:4841] TS 48a83c1507f5482bbb9bc61ac156910d (127.15.222.67:43025): tablet f387cc705cdd483abaaca5546591c2e0 (table test-table [id=754dcf21be6844549d2252a95ee8ec08]) successfully deleted
I20260809 03:34:10.748880 14572 ts_tablet_manager.cc:1905] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:10.751279 14572 ts_tablet_manager.cc:1918] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:10.751369 14572 log.cc:1196] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786246200571663-16249-0/minicluster-data/ts-1/wal/wals/f387cc705cdd483abaaca5546591c2e0
I20260809 03:34:10.751621 14572 ts_tablet_manager.cc:1939] T f387cc705cdd483abaaca5546591c2e0 P 3660cec051a742b1b207f68e10839f5d: Deleting consensus metadata
I20260809 03:34:10.752642 13765 catalog_manager.cc:4841] TS 3660cec051a742b1b207f68e10839f5d (127.15.222.66:33617): tablet f387cc705cdd483abaaca5546591c2e0 (table test-table [id=754dcf21be6844549d2252a95ee8ec08]) successfully deleted
W20260809 03:34:10.755224 13763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:39425 (error 108)
I20260809 03:34:10.756104 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 14100
I20260809 03:34:10.756284 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.755952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:39425 (local address 127.0.0.1:58572)
0809 03:34:10.756129 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:10.756199 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260809 03:34:10.756979 13763 catalog_manager.cc:4584] TS 40d01a9631ed4353bb6955778c6fd7f8 (127.15.222.65:39425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f387cc705cdd483abaaca5546591c2e0: Network error: Client connection negotiation failed: client connection to 127.15.222.65:39425: connect: Connection refused (error 111)
I20260809 03:34:10.768049 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 14301
I20260809 03:34:10.778941 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 13738
2026-08-09T03:34:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2433 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T03:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:10Z Disabled control of system clock
I20260809 03:34:10.812824 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38867
--webserver_interface=127.15.222.126
--webserver_port=0
--builtin_ntp_servers=127.15.222.84:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:10.922544 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:10.922788 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:10.922821 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:10.926712 14584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:10.926765 14584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:10.926780 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:10.926797 14584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:10.926826 14584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:10.931463 14584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:38867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:10.932432 14584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:10.933779 14584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:10.942135 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.942333 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:10.942557 14598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:10.942687 14584 server_base.cc:1034] running on GCE node
I20260809 03:34:10.943320 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:10.944378 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:10.945533 14584 hybrid_clock.cc:648] HybridClock initialized: now 1786246450945513 us; error 31 us; skew 500 ppm
I20260809 03:34:10.947400 14584 webserver.cc:460] Webserver started at http://127.15.222.126:39909/ using document root <none> and password file <none>
I20260809 03:34:10.947953 14584 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:10.948015 14584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:10.948217 14584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:10.950263 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "75f117c1a61740cd99f8bdc077c94dd8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "d7e2481b276c5b14b6f240523960a285"
server_key_iv: "ab41f32e61673f02e13ea339701fe705"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.950790 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "75f117c1a61740cd99f8bdc077c94dd8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "d7e2481b276c5b14b6f240523960a285"
server_key_iv: "ab41f32e61673f02e13ea339701fe705"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.954243 14584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:10.956454 14606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:10.957638 14584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:10.957780 14584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "75f117c1a61740cd99f8bdc077c94dd8"
format_stamp: "Formatted at 2026-08-09 03:34:10 on dist-test-slave-cvwc"
server_key: "d7e2481b276c5b14b6f240523960a285"
server_key_iv: "ab41f32e61673f02e13ea339701fe705"
server_key_version: "encryptionkey@0"
I20260809 03:34:10.957870 14584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:10.972404 14584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:10.973060 14584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:10.973229 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:10.979647 14584 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:38867
I20260809 03:34:10.979657 14683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:38867 every 8 connection(s)
I20260809 03:34:10.980566 14584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:10.983644 14684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:10.987772 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 14584
I20260809 03:34:10.987884 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:10.988137 16249 external_mini_cluster.cc:1428] Setting key fdc862310d46713e9cd86a78134a88af
I20260809 03:34:10.989192 14684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: Bootstrap starting.
I20260809 03:34:10.991504 14684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:10.992259 14684 log.cc:824] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:10.993990 14684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: No bootstrap required, opened a new log
I20260809 03:34:10.996651 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:10.989412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.15.222.126:38867)
0809 03:34:10.990021 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:10.990037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:10.990073 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:34:10.990745 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:10.990760 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:10.991142 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:10.991412 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:10.991801 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.992789 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.993658 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:10.994176 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:10.994319 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:10.995094 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:10.995099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:10.995110 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:10.995465 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:10.995517 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:10.995537 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:10.995621 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:10.995757 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 03:34:10.996000 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260809 03:34:10.997372 14684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } }
I20260809 03:34:10.997614 14684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:10.997673 14684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f117c1a61740cd99f8bdc077c94dd8, State: Initialized, Role: FOLLOWER
I20260809 03:34:10.998133 14684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } }
I20260809 03:34:10.998286 14684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:10.998346 14684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:10.998447 14684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:10.999295 14684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } }
I20260809 03:34:10.999588 14684 leader_election.cc:302] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f117c1a61740cd99f8bdc077c94dd8; no voters:
I20260809 03:34:10.999812 14684 leader_election.cc:288] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:10.999918 14690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:11.000118 14690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 75f117c1a61740cd99f8bdc077c94dd8, State: Running, Role: LEADER
I20260809 03:34:11.000413 14690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } }
I20260809 03:34:11.000840 14684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:11.002345 14694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f117c1a61740cd99f8bdc077c94dd8. Latest consensus state: current_term: 1 leader_uuid: "75f117c1a61740cd99f8bdc077c94dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } } }
I20260809 03:34:11.002368 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f117c1a61740cd99f8bdc077c94dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f117c1a61740cd99f8bdc077c94dd8" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 38867 } } }
I20260809 03:34:11.002493 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:11.002491 14694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:11.002847 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:11.006237 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:11.011428 14714 catalog_manager.cc:1347] Generated new cluster ID: 4242535638ff4cf4aa11df652e0eb872
I20260809 03:34:11.011485 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:11.030515 14714 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:11.031447 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:11.043069 14714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: Generated new TSK 0
I20260809 03:34:11.043706 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:11.054997 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.65:0
--local_ip_for_outbound_sockets=127.15.222.65
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--builtin_ntp_servers=127.15.222.84:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 03:34:11.164134 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:11.164357 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:11.164391 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:11.168195 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:11.168241 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:11.168290 14724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.65
I20260809 03:34:11.172772 14724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.222.65
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.65
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:11.173660 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:11.174971 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:11.184012 14732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:11.184046 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:11.184039 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:11.184355 14724 server_base.cc:1034] running on GCE node
I20260809 03:34:11.184751 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:11.185334 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:11.186486 14724 hybrid_clock.cc:648] HybridClock initialized: now 1786246451186464 us; error 34 us; skew 500 ppm
I20260809 03:34:11.188153 14724 webserver.cc:460] Webserver started at http://127.15.222.65:39055/ using document root <none> and password file <none>
I20260809 03:34:11.188688 14724 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:11.188752 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:11.188949 14724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:11.190976 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data/instance:
uuid: "7fa7b9231985483e8a53344703e30045"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "6311c7168bb7b3521dee0366a4a0fa75"
server_key_iv: "a22b63d90394db79fb456b73489887d5"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.191522 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance:
uuid: "7fa7b9231985483e8a53344703e30045"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "6311c7168bb7b3521dee0366a4a0fa75"
server_key_iv: "a22b63d90394db79fb456b73489887d5"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.194912 14724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:11.197072 14747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.198192 14724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:11.198294 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal
uuid: "7fa7b9231985483e8a53344703e30045"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "6311c7168bb7b3521dee0366a4a0fa75"
server_key_iv: "a22b63d90394db79fb456b73489887d5"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.198382 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:11.207465 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:11.208026 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:11.208189 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:11.208699 14724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:11.209568 14724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:11.209615 14724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.209666 14724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:11.209694 14724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.217921 14724 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.65:40197
I20260809 03:34:11.217941 14917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.65:40197 every 8 connection(s)
I20260809 03:34:11.218741 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/data/info.pb
I20260809 03:34:11.221489 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 14724
I20260809 03:34:11.221617 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-0/wal/instance
I20260809 03:34:11.221982 16249 external_mini_cluster.cc:1428] Setting key 493bed3ca19d997837c4294c8e8ad05f
I20260809 03:34:11.223984 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.66:0
--local_ip_for_outbound_sockets=127.15.222.66
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--builtin_ntp_servers=127.15.222.84:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:11.227488 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.220351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.65:60427 (local address 127.15.222.126:38867)
0809 03:34:11.220475 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.220480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.221485 (+ 1005us) server_negotiation.cc:407] Connection header received
0809 03:34:11.223122 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.223125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.223166 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.223223 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:11.224120 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.224661 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.225760 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.225995 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.226084 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.227053 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:11.227057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:11.227059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:11.227073 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:11.227093 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:11.227100 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:11.227156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.227291 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.227349 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:34:11.228330 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.220683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38867 (local address 127.15.222.65:60427)
0809 03:34:11.221309 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.221362 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.222906 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.223401 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.223410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.223514 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:11.223983 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.223996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.224774 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.224777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.225627 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.225636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.226129 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.226132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.226279 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.226815 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:11.226817 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:11.226855 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:11.226915 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:11.227166 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:11.227176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:11.227183 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.227283 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.227552 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260809 03:34:11.229439 14918 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38867
I20260809 03:34:11.229676 14918 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:11.230300 14918 heartbeater.cc:511] Master 127.15.222.126:38867 requested a full tablet report, sending...
I20260809 03:34:11.231762 14635 ts_manager.cc:194] Registered new tserver with Master: 7fa7b9231985483e8a53344703e30045 (127.15.222.65:40197)
I20260809 03:34:11.232864 14635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.65:60427
W20260809 03:34:11.332608 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:11.332840 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:11.332875 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:11.336753 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:11.336799 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:11.336862 14923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.66
I20260809 03:34:11.341522 14923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.222.66
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.66
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:11.342466 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:11.343722 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:11.352542 14931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:11.352447 14929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:11.352566 14923 server_base.cc:1034] running on GCE node
W20260809 03:34:11.352469 14933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:11.353046 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:11.353622 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:11.354773 14923 hybrid_clock.cc:648] HybridClock initialized: now 1786246451354757 us; error 28 us; skew 500 ppm
I20260809 03:34:11.356379 14923 webserver.cc:460] Webserver started at http://127.15.222.66:41997/ using document root <none> and password file <none>
I20260809 03:34:11.356895 14923 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:11.356959 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:11.357152 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:11.359068 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data/instance:
uuid: "c15b2c1b352a443987431d8df2dfd451"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "01ca7049bce7d56a34b8d389ce0fc1ea"
server_key_iv: "ad3d7bddf9479dd5dc9b6ad17863915a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.359578 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance:
uuid: "c15b2c1b352a443987431d8df2dfd451"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "01ca7049bce7d56a34b8d389ce0fc1ea"
server_key_iv: "ad3d7bddf9479dd5dc9b6ad17863915a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.362979 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:11.365127 14941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.366241 14923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:11.366339 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal
uuid: "c15b2c1b352a443987431d8df2dfd451"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "01ca7049bce7d56a34b8d389ce0fc1ea"
server_key_iv: "ad3d7bddf9479dd5dc9b6ad17863915a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.366425 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:11.378690 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:11.379272 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:11.379429 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:11.379949 14923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:11.380846 14923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:11.380900 14923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.380954 14923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:11.380976 14923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.389039 14923 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.66:42141
I20260809 03:34:11.389061 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.66:42141 every 8 connection(s)
I20260809 03:34:11.389873 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/data/info.pb
I20260809 03:34:11.390095 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 14923
I20260809 03:34:11.390214 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-1/wal/instance
I20260809 03:34:11.390542 16249 external_mini_cluster.cc:1428] Setting key 2be05a6396cdff401e92f9a3e425ebc0
I20260809 03:34:11.392673 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.222.67:0
--local_ip_for_outbound_sockets=127.15.222.67
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--builtin_ntp_servers=127.15.222.84:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 03:34:11.398751 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.391621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.66:42231 (local address 127.15.222.126:38867)
0809 03:34:11.391758 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.391762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.392879 (+ 1117us) server_negotiation.cc:407] Connection header received
0809 03:34:11.394324 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.394327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.394380 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.394458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:11.395304 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.395824 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.396933 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.397147 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.397234 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.398289 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:11.398293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:11.398295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:11.398308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:11.398328 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:11.398335 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:11.398393 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.398549 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.398612 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 03:34:11.399508 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.392069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38867 (local address 127.15.222.66:42231)
0809 03:34:11.392730 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.392783 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.394128 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.394628 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.394635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.394715 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:11.395170 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.395183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.395947 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.395950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.396807 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.396816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.397282 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.397286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.397449 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.398053 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:11.398055 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:11.398091 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:11.398151 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:11.398414 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:11.398422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:11.398427 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.398539 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.398792 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":136,"threads_started":1}
I20260809 03:34:11.401525 15104 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38867
I20260809 03:34:11.401815 15104 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:11.402451 15104 heartbeater.cc:511] Master 127.15.222.126:38867 requested a full tablet report, sending...
I20260809 03:34:11.403530 14635 ts_manager.cc:194] Registered new tserver with Master: c15b2c1b352a443987431d8df2dfd451 (127.15.222.66:42141)
I20260809 03:34:11.404291 14635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.66:42231
W20260809 03:34:11.505507 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:11.505748 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:11.505801 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:11.509649 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:11.509692 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:11.509774 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.222.67
I20260809 03:34:11.514314 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.222.67
--webserver_port=0
--tserver_master_addrs=127.15.222.126:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.222.67
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:11.515225 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:11.516538 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:11.525456 15119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:11.525468 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:11.525467 15121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:11.525513 15108 server_base.cc:1034] running on GCE node
I20260809 03:34:11.526074 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:11.526646 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:11.527801 15108 hybrid_clock.cc:648] HybridClock initialized: now 1786246451527786 us; error 27 us; skew 500 ppm
I20260809 03:34:11.532415 15108 webserver.cc:460] Webserver started at http://127.15.222.67:43519/ using document root <none> and password file <none>
I20260809 03:34:11.532946 15108 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:11.533013 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:11.533217 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:11.535180 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data/instance:
uuid: "bb6c4f65e6de4610995e9f76484be923"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "24146afd81a17b2b21a7c3b5ade18a84"
server_key_iv: "7c4d6004fb25fce57167cb6b0e1c421a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.535691 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance:
uuid: "bb6c4f65e6de4610995e9f76484be923"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "24146afd81a17b2b21a7c3b5ade18a84"
server_key_iv: "7c4d6004fb25fce57167cb6b0e1c421a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.539134 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 03:34:11.541281 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.542399 15108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:11.542510 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal
uuid: "bb6c4f65e6de4610995e9f76484be923"
format_stamp: "Formatted at 2026-08-09 03:34:11 on dist-test-slave-cvwc"
server_key: "24146afd81a17b2b21a7c3b5ade18a84"
server_key_iv: "7c4d6004fb25fce57167cb6b0e1c421a"
server_key_version: "encryptionkey@0"
I20260809 03:34:11.542614 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:11.568048 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:11.568729 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:11.568922 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:11.569531 15108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 03:34:11.570521 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 03:34:11.570569 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.570624 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 03:34:11.570654 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:11.578817 15108 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.67:34347
I20260809 03:34:11.578842 15305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.67:34347 every 8 connection(s)
I20260809 03:34:11.579707 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/data/info.pb
I20260809 03:34:11.587881 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.581375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:58611 (local address 127.15.222.126:38867)
0809 03:34:11.581538 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.581543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.582612 (+ 1069us) server_negotiation.cc:407] Connection header received
0809 03:34:11.583900 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.583903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.583949 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.584016 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:11.584797 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.585298 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.586341 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.586552 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.586620 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.587550 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:11.587552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:11.587554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:11.587563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:11.587577 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:11.587584 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:11.587625 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.587728 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.587770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 03:34:11.588644 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.581819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:38867 (local address 127.15.222.67:58611)
0809 03:34:11.582469 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.582508 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.583732 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.584149 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.584155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.584234 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:11.584695 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.584707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.585379 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.585381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.586259 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.586268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.586655 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.586657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.586790 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.587332 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 03:34:11.587333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:11.587371 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 03:34:11.587433 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:11.587635 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:11.587639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:11.587642 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.587724 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.587938 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260809 03:34:11.589488 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 15108
I20260809 03:34:11.589586 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal/instance
I20260809 03:34:11.589653 15306 heartbeater.cc:347] Connected to a master server at 127.15.222.126:38867
I20260809 03:34:11.589912 15306 heartbeater.cc:464] Registering TS with master...
I20260809 03:34:11.589916 16249 external_mini_cluster.cc:1428] Setting key 0e3e40d7ab8b51010b8de99f87cba0ae
I20260809 03:34:11.590420 15306 heartbeater.cc:511] Master 127.15.222.126:38867 requested a full tablet report, sending...
I20260809 03:34:11.591436 14632 ts_manager.cc:194] Registered new tserver with Master: bb6c4f65e6de4610995e9f76484be923 (127.15.222.67:34347)
I20260809 03:34:11.592077 14632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.222.67:58611
I20260809 03:34:11.603338 16249 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 03:34:11.608577 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.605090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.15.222.126:38867)
0809 03:34:11.605184 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.605189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.605397 (+ 208us) server_negotiation.cc:407] Connection header received
0809 03:34:11.605496 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.605498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.605536 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.605596 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:11.605895 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.606370 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.607183 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.607407 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.607492 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.608214 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:11.608217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:11.608219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:11.608229 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:11.608246 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:11.608253 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:11.608309 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.608401 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.608452 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 03:34:11.611150 14632 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 03:34:11.613977 14632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 03:34:11.627588 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.622627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:40197 (local address 127.0.0.1:51956)
0809 03:34:11.622976 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.622988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.623065 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.623781 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.623784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.623801 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:11.623872 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.623878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.624769 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.624772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.626047 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.626057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.627066 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.627188 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627236 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.627338 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.627443 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260809 03:34:11.627625 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.623046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.15.222.65:40197)
0809 03:34:11.623351 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.623361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.623380 (+ 19us) server_negotiation.cc:407] Connection header received
0809 03:34:11.623447 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.623452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.623632 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.623812 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:11.623953 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.624664 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.626175 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.626940 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627245 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.627368 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.627466 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260809 03:34:11.627942 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.621971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42141 (local address 127.0.0.1:54870)
0809 03:34:11.622824 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.622837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.622946 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.623477 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.623482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.623506 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:11.623640 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.623648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.625044 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.625047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.626388 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.626398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.627554 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.627647 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627671 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.627740 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.627816 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":246,"thread_start_us":103,"threads_started":1}
I20260809 03:34:11.628105 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.621822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.67:34347 (local address 127.0.0.1:37024)
0809 03:34:11.622577 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.622625 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.622746 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.623618 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.623621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.623638 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:11.623726 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.623732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.625314 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.625317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.626409 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.626416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.627165 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.627249 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627266 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.627314 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.627983 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":620,"thread_start_us":89,"threads_started":1}
I20260809 03:34:11.628412 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.622150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.15.222.66:42141)
0809 03:34:11.622694 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.622706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.623040 (+ 334us) server_negotiation.cc:407] Connection header received
0809 03:34:11.623123 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.623129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.623324 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.623472 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:11.623761 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.624933 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.626501 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.627212 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627310 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627489 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.628155 (+ 666us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.628253 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":123,"threads_started":1}
I20260809 03:34:11.630429 14819 tablet_service.cc:1467] Processing CreateTablet for tablet f50537821fee4ce09efeb8b53205cfe0 (DEFAULT_TABLE table=test-table [id=3d6038bb92a94acfa39c4e683ad4753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:11.630638 15011 tablet_service.cc:1467] Processing CreateTablet for tablet f50537821fee4ce09efeb8b53205cfe0 (DEFAULT_TABLE table=test-table [id=3d6038bb92a94acfa39c4e683ad4753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:11.631220 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.621940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.15.222.67:34347)
0809 03:34:11.623197 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.623210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.623231 (+ 21us) server_negotiation.cc:407] Connection header received
0809 03:34:11.623293 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.623297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.623469 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.623881 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:11.624012 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.625095 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.626522 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.627038 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.627435 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.627608 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.627658 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.631033 (+ 3375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":123,"threads_started":1}
I20260809 03:34:11.631511 14819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50537821fee4ce09efeb8b53205cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:11.631596 15011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50537821fee4ce09efeb8b53205cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:11.633850 15211 tablet_service.cc:1467] Processing CreateTablet for tablet f50537821fee4ce09efeb8b53205cfe0 (DEFAULT_TABLE table=test-table [id=3d6038bb92a94acfa39c4e683ad4753a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 03:34:11.634805 15211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50537821fee4ce09efeb8b53205cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:11.637761 15330 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Bootstrap starting.
I20260809 03:34:11.639024 15331 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Bootstrap starting.
I20260809 03:34:11.639856 15330 tablet_bootstrap.cc:654] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:11.640545 15330 log.cc:824] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:11.641152 15331 tablet_bootstrap.cc:654] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:11.641404 15333 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Bootstrap starting.
I20260809 03:34:11.641989 15331 log.cc:824] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:11.642664 15330 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: No bootstrap required, opened a new log
I20260809 03:34:11.642834 15330 ts_tablet_manager.cc:1397] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:34:11.643493 15333 tablet_bootstrap.cc:654] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:11.643807 15331 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: No bootstrap required, opened a new log
I20260809 03:34:11.643939 15331 ts_tablet_manager.cc:1397] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 03:34:11.644244 15333 log.cc:824] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:11.646202 15333 tablet_bootstrap.cc:492] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: No bootstrap required, opened a new log
I20260809 03:34:11.646380 15333 ts_tablet_manager.cc:1397] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 03:34:11.647001 15330 raft_consensus.cc:348] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.647262 15330 raft_consensus.cc:374] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:11.647162 15331 raft_consensus.cc:348] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.647320 15330 raft_consensus.cc:729] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15b2c1b352a443987431d8df2dfd451, State: Initialized, Role: FOLLOWER
I20260809 03:34:11.647374 15331 raft_consensus.cc:374] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:11.647439 15331 raft_consensus.cc:729] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7b9231985483e8a53344703e30045, State: Initialized, Role: FOLLOWER
I20260809 03:34:11.647751 15330 consensus_queue.cc:260] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.647863 15331 consensus_queue.cc:260] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.648399 15104 heartbeater.cc:503] Master 127.15.222.126:38867 was elected leader, sending a full tablet report...
I20260809 03:34:11.648722 15330 ts_tablet_manager.cc:1428] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:34:11.649957 15331 ts_tablet_manager.cc:1428] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 03:34:11.650293 14918 heartbeater.cc:503] Master 127.15.222.126:38867 was elected leader, sending a full tablet report...
I20260809 03:34:11.650141 15333 raft_consensus.cc:348] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.650374 15333 raft_consensus.cc:374] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:11.650432 15333 raft_consensus.cc:729] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6c4f65e6de4610995e9f76484be923, State: Initialized, Role: FOLLOWER
I20260809 03:34:11.650816 15333 consensus_queue.cc:260] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.651634 15333 ts_tablet_manager.cc:1428] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 03:34:11.651947 15306 heartbeater.cc:503] Master 127.15.222.126:38867 was elected leader, sending a full tablet report...
W20260809 03:34:11.719774 14919 tablet.cc:2369] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:11.747056 15338 raft_consensus.cc:482] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 03:34:11.747244 15338 raft_consensus.cc:504] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.748361 15338 leader_election.cc:288] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c15b2c1b352a443987431d8df2dfd451 (127.15.222.66:42141), 7fa7b9231985483e8a53344703e30045 (127.15.222.65:40197)
I20260809 03:34:11.752555 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.748788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:57907 (local address 127.15.222.66:42141)
0809 03:34:11.748913 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.748917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.749122 (+ 205us) server_negotiation.cc:407] Connection header received
0809 03:34:11.749198 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.749200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.749243 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.749318 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:11.749589 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.750256 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.751541 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.752103 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.752198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.752239 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.752373 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.752437 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:34:11.752556 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.748676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.222.67:39841 (local address 127.15.222.65:40197)
0809 03:34:11.748795 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.748799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.748817 (+ 18us) server_negotiation.cc:407] Connection header received
0809 03:34:11.748872 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.748875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.748927 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.749006 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 03:34:11.749292 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.750036 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.751448 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.752014 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.752129 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.752186 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.752346 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.752424 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 03:34:11.752584 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.748573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:40197 (local address 127.15.222.67:39841)
0809 03:34:11.748675 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.748689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.748776 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.749035 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.749038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.749061 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:11.749186 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.749193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.750133 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.750135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.751325 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.751337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.752118 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.752121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.752227 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.752254 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.752332 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.752430 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":6}
I20260809 03:34:11.752590 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.748573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42141 (local address 127.15.222.67:57907)
0809 03:34:11.749021 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.749033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:11.749121 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:11.749341 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:11.749343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:11.749362 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 03:34:11.749479 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.749485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.750347 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.750350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.751447 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:11.751455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.752194 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.752197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:11.752284 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.752302 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 03:34:11.752360 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 03:34:11.752441 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":90,"thread_start_us":138,"threads_started":1}
I20260809 03:34:11.753304 14845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50537821fee4ce09efeb8b53205cfe0" candidate_uuid: "bb6c4f65e6de4610995e9f76484be923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b9231985483e8a53344703e30045" is_pre_election: true
I20260809 03:34:11.753310 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50537821fee4ce09efeb8b53205cfe0" candidate_uuid: "bb6c4f65e6de4610995e9f76484be923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15b2c1b352a443987431d8df2dfd451" is_pre_election: true
I20260809 03:34:11.753621 15041 raft_consensus.cc:2454] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6c4f65e6de4610995e9f76484be923 in term 0.
I20260809 03:34:11.753621 14845 raft_consensus.cc:2454] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6c4f65e6de4610995e9f76484be923 in term 0.
I20260809 03:34:11.754158 15136 leader_election.cc:302] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb6c4f65e6de4610995e9f76484be923, c15b2c1b352a443987431d8df2dfd451; no voters:
I20260809 03:34:11.754432 15338 raft_consensus.cc:2781] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 03:34:11.754513 15338 raft_consensus.cc:482] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 03:34:11.754562 15338 raft_consensus.cc:3037] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:11.755553 15338 raft_consensus.cc:504] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.755870 15338 leader_election.cc:288] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [CANDIDATE]: Term 1 election: Requested vote from peers c15b2c1b352a443987431d8df2dfd451 (127.15.222.66:42141), 7fa7b9231985483e8a53344703e30045 (127.15.222.65:40197)
I20260809 03:34:11.756196 14845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50537821fee4ce09efeb8b53205cfe0" candidate_uuid: "bb6c4f65e6de4610995e9f76484be923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b9231985483e8a53344703e30045"
I20260809 03:34:11.756196 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50537821fee4ce09efeb8b53205cfe0" candidate_uuid: "bb6c4f65e6de4610995e9f76484be923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15b2c1b352a443987431d8df2dfd451"
I20260809 03:34:11.756315 14845 raft_consensus.cc:3037] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:11.756315 15041 raft_consensus.cc:3037] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:11.757544 14845 raft_consensus.cc:2454] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6c4f65e6de4610995e9f76484be923 in term 1.
I20260809 03:34:11.757544 15041 raft_consensus.cc:2454] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6c4f65e6de4610995e9f76484be923 in term 1.
I20260809 03:34:11.757867 15134 leader_election.cc:302] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fa7b9231985483e8a53344703e30045, bb6c4f65e6de4610995e9f76484be923, c15b2c1b352a443987431d8df2dfd451; no voters:
I20260809 03:34:11.758076 15338 raft_consensus.cc:2781] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:11.758244 15338 raft_consensus.cc:686] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 1 LEADER]: Becoming Leader. State: Replica: bb6c4f65e6de4610995e9f76484be923, State: Running, Role: LEADER
I20260809 03:34:11.758554 15338 consensus_queue.cc:237] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } }
I20260809 03:34:11.761729 14632 catalog_manager.cc:5495] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6c4f65e6de4610995e9f76484be923 (127.15.222.67). New cstate: current_term: 1 leader_uuid: "bb6c4f65e6de4610995e9f76484be923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6c4f65e6de4610995e9f76484be923" member_type: VOTER last_known_addr { host: "127.15.222.67" port: 34347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20260809 03:34:11.776369 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.773509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.15.222.67:34347)
0809 03:34:11.773605 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:11.773608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:11.773620 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:11.773749 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:11.773752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:11.773803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:11.773893 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:11.774062 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.774527 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.775260 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:11.775460 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:11.775527 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:11.775730 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:11.776111 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:11.776192 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:11.776217 (+ 25us) server_negotiation.cc:299] Negotiation successful
0809 03:34:11.776258 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:34:11.781823 14845 raft_consensus.cc:1261] T f50537821fee4ce09efeb8b53205cfe0 P 7fa7b9231985483e8a53344703e30045 [term 1 FOLLOWER]: Refusing update from remote peer bb6c4f65e6de4610995e9f76484be923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:11.781822 15041 raft_consensus.cc:1261] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 1 FOLLOWER]: Refusing update from remote peer bb6c4f65e6de4610995e9f76484be923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:11.782550 15341 consensus_queue.cc:1035] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa7b9231985483e8a53344703e30045" member_type: VOTER last_known_addr { host: "127.15.222.65" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:11.782711 15338 consensus_queue.cc:1035] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15b2c1b352a443987431d8df2dfd451" member_type: VOTER last_known_addr { host: "127.15.222.66" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:11.791311 15347 mvcc.cc:205] Tried to move back new op lower bound from 7316465466491527168 to 7316465466405613568. Current Snapshot: MvccSnapshot[applied={T|T < 7316465466491527168}]
I20260809 03:34:11.798626 14632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:11.798789 14632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 03:34:11.801033 14632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75f117c1a61740cd99f8bdc077c94dd8: Sending DeleteTablet for 3 replicas of tablet f50537821fee4ce09efeb8b53205cfe0
I20260809 03:34:11.801776 15011 tablet_service.cc:1514] Processing DeleteTablet for tablet f50537821fee4ce09efeb8b53205cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:11 UTC) from {username='slave'} at 127.0.0.1:54870
I20260809 03:34:11.801846 14819 tablet_service.cc:1514] Processing DeleteTablet for tablet f50537821fee4ce09efeb8b53205cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:11 UTC) from {username='slave'} at 127.0.0.1:51956
I20260809 03:34:11.802249 15360 tablet_replica.cc:330] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: stopping tablet replica
I20260809 03:34:11.802300 15211 tablet_service.cc:1514] Processing DeleteTablet for tablet f50537821fee4ce09efeb8b53205cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 03:34:11 UTC) from {username='slave'} at 127.0.0.1:37024
I20260809 03:34:11.802673 15360 raft_consensus.cc:2229] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 03:34:11.802886 15363 tablet_replica.cc:330] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: stopping tablet replica
I20260809 03:34:11.803092 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 14724
I20260809 03:34:11.803198 15360 raft_consensus.cc:2258] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:11.807350 15363 raft_consensus.cc:2229] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 1 LEADER]: Raft consensus shutting down.
I20260809 03:34:11.807722 15363 raft_consensus.cc:2258] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 03:34:11.810431 15363 ts_tablet_manager.cc:1905] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 03:34:11.813097 15363 ts_tablet_manager.cc:1918] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 03:34:11.813189 15363 log.cc:1196] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Deleting WAL directory at /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786246200571663-16249-0/minicluster-data/ts-2/wal/wals/f50537821fee4ce09efeb8b53205cfe0
I20260809 03:34:11.813498 15363 ts_tablet_manager.cc:1939] T f50537821fee4ce09efeb8b53205cfe0 P bb6c4f65e6de4610995e9f76484be923: Deleting consensus metadata
I20260809 03:34:11.814622 14609 catalog_manager.cc:4841] TS bb6c4f65e6de4610995e9f76484be923 (127.15.222.67:34347): tablet f50537821fee4ce09efeb8b53205cfe0 (table test-table [id=3d6038bb92a94acfa39c4e683ad4753a]) successfully deleted
I20260809 03:34:11.817777 15360 ts_tablet_manager.cc:1905] T f50537821fee4ce09efeb8b53205cfe0 P c15b2c1b352a443987431d8df2dfd451: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 03:34:11.818682 14609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.222.65:40197 (error 108)
I20260809 03:34:11.819595 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.819322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.65:40197 (local address 127.0.0.1:51972)
0809 03:34:11.819446 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.819513 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.65:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 03:34:11.819693 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 14923
W20260809 03:34:11.822526 14609 catalog_manager.cc:4584] TS 7fa7b9231985483e8a53344703e30045 (127.15.222.65:40197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f50537821fee4ce09efeb8b53205cfe0: Network error: Client connection negotiation failed: client connection to 127.15.222.65:40197: connect: Connection refused (error 111)
I20260809 03:34:11.827457 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 15108
I20260809 03:34:11.827623 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:11.827362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.66:42141 (local address 127.0.0.1:54886)
0809 03:34:11.827492 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:11.827549 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.66:42141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:34:11.844412 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 14584
2026-08-09T03:34:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1066 ms)
[----------] 6 tests from EncryptionPolicyTest (10598 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:11 dist-test-slave-cvwc krb5kdc[15369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:11 dist-test-slave-cvwc krb5kdc[15369](info): set up 2 sockets
Aug 09 03:34:11 dist-test-slave-cvwc krb5kdc[15369](info): commencing operation
krb5kdc: starting...
W20260809 03:34:12.981462 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:13.100586 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:13.210316 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:13.210572 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:13.210619 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:13.214542 15390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:13.214599 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:13.214617 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:13.214632 15390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:13.214648 15390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:13.219950 15390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:13.220934 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:13.221954 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:34:13.230083 15390 server_base.cc:1034] running on GCE node
W20260809 03:34:13.230007 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:13.230007 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:13.230027 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:13.230916 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:13.231998 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:13.233160 15390 hybrid_clock.cc:648] HybridClock initialized: now 1786246453233147 us; error 27 us; skew 500 ppm
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:13.235886 15390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:13.236773 15390 webserver.cc:460] Webserver started at http://127.0.0.1:40373/ using document root <none> and password file <none>
I20260809 03:34:13.237298 15390 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:13.237365 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:13.237581 15390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:13.239619 15390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "0d96f32f340d4609b77794056976bc67"
format_stamp: "Formatted at 2026-08-09 03:34:13 on dist-test-slave-cvwc"
server_key: "d05a5ddf63cd5da3865f6ebe009c0cb7"
server_key_iv: "736a4871fb36c44e2a841d213729e22a"
server_key_version: "encryptionkey@0"
I20260809 03:34:13.240160 15390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "0d96f32f340d4609b77794056976bc67"
format_stamp: "Formatted at 2026-08-09 03:34:13 on dist-test-slave-cvwc"
server_key: "d05a5ddf63cd5da3865f6ebe009c0cb7"
server_key_iv: "736a4871fb36c44e2a841d213729e22a"
server_key_version: "encryptionkey@0"
I20260809 03:34:13.243737 15390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:13.245992 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:13.247105 15390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:13.247227 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "0d96f32f340d4609b77794056976bc67"
format_stamp: "Formatted at 2026-08-09 03:34:13 on dist-test-slave-cvwc"
server_key: "d05a5ddf63cd5da3865f6ebe009c0cb7"
server_key_iv: "736a4871fb36c44e2a841d213729e22a"
server_key_version: "encryptionkey@0"
I20260809 03:34:13.247314 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:13.286485 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:13.289318 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:13.289505 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:13.296113 15390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46359
I20260809 03:34:13.296160 15492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46359 every 8 connection(s)
I20260809 03:34:13.297053 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:13.299773 15493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:13.305238 15493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67: Bootstrap starting.
I20260809 03:34:13.306535 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 15390
I20260809 03:34:13.306658 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:13.306941 16249 external_mini_cluster.cc:1428] Setting key fa7077f549e77789ac7544942ab6269d
I20260809 03:34:13.307986 15493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:13.308787 15493 log.cc:824] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:13.310658 15493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67: No bootstrap required, opened a new log
I20260809 03:34:13.314044 15493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } }
I20260809 03:34:13.314260 15493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:13.314327 15493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d96f32f340d4609b77794056976bc67, State: Initialized, Role: FOLLOWER
I20260809 03:34:13.314766 15493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } }
I20260809 03:34:13.314939 15493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:13.315006 15493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:13.315111 15493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:34:13.315976 15493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } }
I20260809 03:34:13.316290 15493 leader_election.cc:302] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d96f32f340d4609b77794056976bc67; no voters:
I20260809 03:34:13.316583 15493 leader_election.cc:288] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:13.316664 15502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:13.317118 15502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [term 1 LEADER]: Becoming Leader. State: Replica: 0d96f32f340d4609b77794056976bc67, State: Running, Role: LEADER
I20260809 03:34:13.317405 15502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } }
I20260809 03:34:13.317885 15493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:13.318555 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d96f32f340d4609b77794056976bc67. Latest consensus state: current_term: 1 leader_uuid: "0d96f32f340d4609b77794056976bc67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } } }
I20260809 03:34:13.318668 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:13.319664 15504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d96f32f340d4609b77794056976bc67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d96f32f340d4609b77794056976bc67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46359 } } }
I20260809 03:34:13.319821 15504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:13.320628 15515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:13.321265 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:13.308073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60157 (local address 127.0.0.1:46359)
0809 03:34:13.308518 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:13.308533 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:13.308568 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:34:13.309277 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:13.309302 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:13.309703 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:13.310046 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:13.310831 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:13.311793 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:13.312617 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:13.313071 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:13.313218 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:13.315896 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:13.315901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:13.315913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:13.315939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:13.317914 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:13.318615 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:13.318621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:13.318627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:13.318766 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:13.319205 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:13.319208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:13.319210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:13.319721 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:13.319950 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:13.320238 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 03:34:13.320470 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260809 03:34:13.323441 15515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:13.324545 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:13.328928 15515 catalog_manager.cc:1347] Generated new cluster ID: 5cef80e6874f4e209e9b9930359a5d29
I20260809 03:34:13.328991 15515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:13.335090 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:13.326544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38083 (local address 127.0.0.1:46359)
0809 03:34:13.326642 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:13.326646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:13.326840 (+ 194us) server_negotiation.cc:407] Connection header received
0809 03:34:13.326953 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:13.326956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:13.326993 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:13.327053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:13.327678 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:13.328867 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:13.330022 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:13.330351 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:13.330448 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:13.331952 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:13.331955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:13.331958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:13.331981 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:13.333485 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:13.333991 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:13.333994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:13.333996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:13.334068 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:13.334343 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:13.334345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:13.334347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:13.334516 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:13.334632 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:13.334859 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 03:34:13.334949 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:34:13.352957 15515 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:13.354276 15515 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:13.361842 15515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d96f32f340d4609b77794056976bc67: Generated new TSK 0
I20260809 03:34:13.362528 15515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:13.373328 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 15390
W20260809 03:34:13.389221 15367 connection.cc:535] client connection to 127.0.0.1:46359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46359): Connection reset by peer (error 104)
2026-08-09T03:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15543](info): set up 2 sockets
Aug 09 03:34:13 dist-test-slave-cvwc krb5kdc[15543](info): commencing operation
krb5kdc: starting...
W20260809 03:34:14.521642 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:14 dist-test-slave-cvwc krb5kdc[15543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:14.636444 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:14.745488 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:14.745748 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:14.745813 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:14.749614 15563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:14.749670 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:14.749696 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:14.749734 15563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:14.749758 15563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:14.754840 15563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:14.755808 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:14.756780 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:14.764822 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:14.764818 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:14.764822 15580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:14.765023 15563 server_base.cc:1034] running on GCE node
I20260809 03:34:14.765761 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:14.766806 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:14.767956 15563 hybrid_clock.cc:648] HybridClock initialized: now 1786246454767944 us; error 25 us; skew 500 ppm
Aug 09 03:34:14 dist-test-slave-cvwc krb5kdc[15543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:14.770550 15563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:14.771356 15563 webserver.cc:460] Webserver started at http://127.0.0.1:43793/ using document root <none> and password file <none>
I20260809 03:34:14.771880 15563 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:14.771950 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:14.772154 15563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:14.774160 15563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "80723e6fd7834ba9a147ef343d119705"
format_stamp: "Formatted at 2026-08-09 03:34:14 on dist-test-slave-cvwc"
server_key: "194816790a197e86ba6c1ff782cf306f"
server_key_iv: "c6844778d1b4cb00cddb746659768620"
server_key_version: "encryptionkey@0"
I20260809 03:34:14.774698 15563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "80723e6fd7834ba9a147ef343d119705"
format_stamp: "Formatted at 2026-08-09 03:34:14 on dist-test-slave-cvwc"
server_key: "194816790a197e86ba6c1ff782cf306f"
server_key_iv: "c6844778d1b4cb00cddb746659768620"
server_key_version: "encryptionkey@0"
I20260809 03:34:14.778126 15563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:14.780375 15590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:14.781560 15563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:14.781679 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "80723e6fd7834ba9a147ef343d119705"
format_stamp: "Formatted at 2026-08-09 03:34:14 on dist-test-slave-cvwc"
server_key: "194816790a197e86ba6c1ff782cf306f"
server_key_iv: "c6844778d1b4cb00cddb746659768620"
server_key_version: "encryptionkey@0"
I20260809 03:34:14.781816 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:14.793170 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:14.795673 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:14.795838 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:14.802348 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43711
I20260809 03:34:14.802361 15662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43711 every 8 connection(s)
I20260809 03:34:14.803242 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:14.805914 15664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:14.811039 15664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705: Bootstrap starting.
I20260809 03:34:14.812312 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 15563
I20260809 03:34:14.812434 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:14.812674 16249 external_mini_cluster.cc:1428] Setting key 33623c53203354ac904635dda8e51a45
I20260809 03:34:14.813474 15664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:14.814392 15664 log.cc:824] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:14.816432 15664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705: No bootstrap required, opened a new log
I20260809 03:34:14.820004 15664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260809 03:34:14.820266 15664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:14.820367 15664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80723e6fd7834ba9a147ef343d119705, State: Initialized, Role: FOLLOWER
I20260809 03:34:14.820823 15664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260809 03:34:14.820956 15664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:14.821000 15664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:14.821094 15664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:14.822042 15664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260809 03:34:14.822420 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:14.813654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52971 (local address 127.0.0.1:43711)
0809 03:34:14.814117 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:14.814135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:14.814176 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:34:14.814944 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:14.814963 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:14.815341 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:14.815682 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:14.816081 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:14.817469 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:14.818330 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:14.819724 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:14.819877 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:14.820722 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:14.820727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:14.820739 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:14.821141 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:14.821208 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:14.821235 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:14.821336 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:14.821529 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 03:34:14.821659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":140,"threads_started":1}
I20260809 03:34:14.822404 15664 leader_election.cc:302] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80723e6fd7834ba9a147ef343d119705; no voters:
I20260809 03:34:14.822731 15664 leader_election.cc:288] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:14.822811 15672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:14.822993 15672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [term 1 LEADER]: Becoming Leader. State: Replica: 80723e6fd7834ba9a147ef343d119705, State: Running, Role: LEADER
I20260809 03:34:14.823325 15672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260809 03:34:14.823956 15664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:14.825158 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:14.825110 15676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80723e6fd7834ba9a147ef343d119705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260809 03:34:14.825114 15675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80723e6fd7834ba9a147ef343d119705. Latest consensus state: current_term: 1 leader_uuid: "80723e6fd7834ba9a147ef343d119705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80723e6fd7834ba9a147ef343d119705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260809 03:34:14.825255 15676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:14.825255 15675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:14.825618 15685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:14.828783 15685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:14.833475 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:14.827366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46727 (local address 127.0.0.1:43711)
0809 03:34:14.827464 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:14.827468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:14.829167 (+ 1699us) server_negotiation.cc:407] Connection header received
0809 03:34:14.829471 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:14.829475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:14.829519 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:14.829585 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:14.829928 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:14.830436 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:14.831468 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:14.831807 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:14.831915 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:14.833055 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:14.833058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:14.833060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:14.833071 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:14.833095 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:14.833105 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:14.833169 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:14.833316 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 03:34:14.833348 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 03:34:14.834993 15685 catalog_manager.cc:1347] Generated new cluster ID: 31dba7df3b494e3eb8192d88d2c08e19
I20260809 03:34:14.835057 15685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:14.842806 15685 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:14.843716 15685 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:14.853669 15685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80723e6fd7834ba9a147ef343d119705: Generated new TSK 0
I20260809 03:34:14.854369 15685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:14.861398 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 15563
W20260809 03:34:14.871711 15541 connection.cc:535] client connection to 127.0.0.1:43711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43711): Connection reset by peer (error 104)
2026-08-09T03:34:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:14 dist-test-slave-cvwc krb5kdc[15711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:14 dist-test-slave-cvwc krb5kdc[15711](info): set up 2 sockets
Aug 09 03:34:14 dist-test-slave-cvwc krb5kdc[15711](info): commencing operation
krb5kdc: starting...
W20260809 03:34:16.003114 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:16 dist-test-slave-cvwc krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:16.117651 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:16.225903 15737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:16.226150 15737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:16.226207 15737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:16.230115 15737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:16.230185 15737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:16.230208 15737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:16.230232 15737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:16.230259 15737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:16.235483 15737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:16.236526 15737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:16.237540 15737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:16.245769 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:16.245759 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:16.245776 15737 server_base.cc:1034] running on GCE node
W20260809 03:34:16.245759 15745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:16.246480 15737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:16.247509 15737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:16.248665 15737 hybrid_clock.cc:648] HybridClock initialized: now 1786246456248651 us; error 25 us; skew 500 ppm
Aug 09 03:34:16 dist-test-slave-cvwc krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:16.251247 15737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:16.252087 15737 webserver.cc:460] Webserver started at http://127.0.0.1:40185/ using document root <none> and password file <none>
I20260809 03:34:16.252622 15737 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:16.252683 15737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:16.252879 15737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:16.254932 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "b6fae97b55f642cea551cc683fc3bc36"
format_stamp: "Formatted at 2026-08-09 03:34:16 on dist-test-slave-cvwc"
server_key: "d0f741ce9905f0466d8f0706f41980b0"
server_key_iv: "aa7cba02e1918cb56a77ee2c50851fe8"
server_key_version: "encryptionkey@0"
I20260809 03:34:16.255445 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "b6fae97b55f642cea551cc683fc3bc36"
format_stamp: "Formatted at 2026-08-09 03:34:16 on dist-test-slave-cvwc"
server_key: "d0f741ce9905f0466d8f0706f41980b0"
server_key_iv: "aa7cba02e1918cb56a77ee2c50851fe8"
server_key_version: "encryptionkey@0"
I20260809 03:34:16.258898 15737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:16.261057 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:16.262220 15737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:16.262324 15737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "b6fae97b55f642cea551cc683fc3bc36"
format_stamp: "Formatted at 2026-08-09 03:34:16 on dist-test-slave-cvwc"
server_key: "d0f741ce9905f0466d8f0706f41980b0"
server_key_iv: "aa7cba02e1918cb56a77ee2c50851fe8"
server_key_version: "encryptionkey@0"
I20260809 03:34:16.262422 15737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:16.266718 15737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:16.269230 15737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:16.269362 15737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:16.275902 15737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34563
I20260809 03:34:16.275918 15834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34563 every 8 connection(s)
I20260809 03:34:16.276808 15737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:16.279482 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:16.282866 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 15737
I20260809 03:34:16.282977 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:16.283233 16249 external_mini_cluster.cc:1428] Setting key fadd6be4b32fda6c47a52d2cde33aa9a
I20260809 03:34:16.285182 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36: Bootstrap starting.
I20260809 03:34:16.287379 15835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:16.288089 15835 log.cc:824] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:16.288259 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:16.284563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35295 (local address 127.0.0.1:34563)
0809 03:34:16.284983 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:16.284998 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:16.285034 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:34:16.285738 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:16.285763 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:16.286045 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:16.286334 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:16.287094 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:16.287098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:16.287110 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:16.287120 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:16.287171 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:16.287190 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:16.287280 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:16.287408 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 03:34:16.287565 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260809 03:34:16.290032 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36: No bootstrap required, opened a new log
I20260809 03:34:16.290825 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:16.294093 15835 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } }
I20260809 03:34:16.294312 15835 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:16.294384 15835 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fae97b55f642cea551cc683fc3bc36, State: Initialized, Role: FOLLOWER
I20260809 03:34:16.294492 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:16.292963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39461 (local address 127.0.0.1:34563)
0809 03:34:16.293073 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:16.293077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:16.293270 (+ 193us) server_negotiation.cc:407] Connection header received
0809 03:34:16.293363 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:16.293366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:16.293395 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:16.293458 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:16.294093 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:16.294096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:16.294099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:16.294109 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:16.294134 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:16.294144 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:16.294211 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:16.294294 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 03:34:16.294328 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 03:34:16.294852 15835 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } }
I20260809 03:34:16.294991 15835 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:16.295051 15835 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:16.295136 15835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:16.295948 15835 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } }
I20260809 03:34:16.296248 15835 leader_election.cc:302] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fae97b55f642cea551cc683fc3bc36; no voters:
I20260809 03:34:16.296530 15835 leader_election.cc:288] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:16.296614 15853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:16.296824 15853 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [term 1 LEADER]: Becoming Leader. State: Replica: b6fae97b55f642cea551cc683fc3bc36, State: Running, Role: LEADER
I20260809 03:34:16.297154 15853 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } }
I20260809 03:34:16.297572 15835 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:16.298627 15855 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fae97b55f642cea551cc683fc3bc36. Latest consensus state: current_term: 1 leader_uuid: "b6fae97b55f642cea551cc683fc3bc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } } }
I20260809 03:34:16.298732 15855 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:16.299108 15865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:16.299078 15854 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fae97b55f642cea551cc683fc3bc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fae97b55f642cea551cc683fc3bc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34563 } } }
I20260809 03:34:16.299206 15854 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:16.302021 15865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:16.307237 15865 catalog_manager.cc:1347] Generated new cluster ID: 44656af0aa4d40d88421c7110b4fa2b4
I20260809 03:34:16.307297 15865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:16.322320 15865 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:16.322625 15865 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:16.331156 15865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fae97b55f642cea551cc683fc3bc36: Generated new TSK 0
I20260809 03:34:16.331867 15865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:16.335529 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 15737
2026-08-09T03:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:16 dist-test-slave-cvwc krb5kdc[15881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:16 dist-test-slave-cvwc krb5kdc[15881](info): set up 2 sockets
Aug 09 03:34:16 dist-test-slave-cvwc krb5kdc[15881](info): commencing operation
krb5kdc: starting...
W20260809 03:34:17.483287 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[15881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:17.596652 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:17.708369 15897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:17.708629 15897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:17.708683 15897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:17.712643 15897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:17.712698 15897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:17.712728 15897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:17.712765 15897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:17.712800 15897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:17.717935 15897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:17.718924 15897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:17.719915 15897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:17.728057 15903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:17.728068 15910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:17.728068 15906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:17.728298 15897 server_base.cc:1034] running on GCE node
I20260809 03:34:17.728866 15897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:17.729972 15897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:17.731141 15897 hybrid_clock.cc:648] HybridClock initialized: now 1786246457731112 us; error 38 us; skew 500 ppm
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[15881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:17.737427 15897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:17.738353 15897 webserver.cc:460] Webserver started at http://127.0.0.1:41849/ using document root <none> and password file <none>
I20260809 03:34:17.738894 15897 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:17.738962 15897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:17.739153 15897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:17.741179 15897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "13955c214ab2450484031930e729b117"
format_stamp: "Formatted at 2026-08-09 03:34:17 on dist-test-slave-cvwc"
server_key: "b7fb724a7f487c9eccafccf6173eccc7"
server_key_iv: "535e15074439c11cc37b16bd0c446495"
server_key_version: "encryptionkey@0"
I20260809 03:34:17.741794 15897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "13955c214ab2450484031930e729b117"
format_stamp: "Formatted at 2026-08-09 03:34:17 on dist-test-slave-cvwc"
server_key: "b7fb724a7f487c9eccafccf6173eccc7"
server_key_iv: "535e15074439c11cc37b16bd0c446495"
server_key_version: "encryptionkey@0"
I20260809 03:34:17.745374 15897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 03:34:17.747742 15917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:17.748916 15897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 03:34:17.749037 15897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "13955c214ab2450484031930e729b117"
format_stamp: "Formatted at 2026-08-09 03:34:17 on dist-test-slave-cvwc"
server_key: "b7fb724a7f487c9eccafccf6173eccc7"
server_key_iv: "535e15074439c11cc37b16bd0c446495"
server_key_version: "encryptionkey@0"
I20260809 03:34:17.749132 15897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:17.767586 15897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:17.770419 15897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:17.770609 15897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:17.777776 15998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37209 every 8 connection(s)
I20260809 03:34:17.777781 15897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37209
I20260809 03:34:17.778805 15897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:17.781656 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:17.782403 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 15897
I20260809 03:34:17.782545 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:17.782861 16249 external_mini_cluster.cc:1428] Setting key 9dd15860556256b4e685e6dc3d14e6ed
I20260809 03:34:17.788307 15999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117: Bootstrap starting.
I20260809 03:34:17.790763 15999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:17.791468 15999 log.cc:824] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117: Log is configured to *not* fsync() on all Append() calls
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[15881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:34:17.793474 15999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117: No bootstrap required, opened a new log
I20260809 03:34:17.797245 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:17.784280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41875 (local address 127.0.0.1:37209)
0809 03:34:17.784717 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:17.784735 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:17.784777 (+ 42us) server_negotiation.cc:407] Connection header received
0809 03:34:17.785582 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:17.785606 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:17.786044 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:17.786406 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:17.787096 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:17.788531 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:17.789370 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:17.790063 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:17.790214 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:17.792708 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:17.792715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:17.792730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:17.792761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:17.794508 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:17.794977 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:17.794981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:17.794985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:17.795095 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:17.795389 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:17.795393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:17.795395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:17.795791 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:17.795931 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:17.796245 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 03:34:17.796525 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260809 03:34:17.797200 15999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260809 03:34:17.797403 15999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:17.797483 15999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13955c214ab2450484031930e729b117, State: Initialized, Role: FOLLOWER
I20260809 03:34:17.797915 15999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260809 03:34:17.798040 15999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:17.798097 15999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:17.798183 15999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:17.799176 15999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260809 03:34:17.799516 15999 leader_election.cc:302] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13955c214ab2450484031930e729b117; no voters:
I20260809 03:34:17.799798 15999 leader_election.cc:288] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:17.799881 16004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:17.800113 16004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [term 1 LEADER]: Becoming Leader. State: Replica: 13955c214ab2450484031930e729b117, State: Running, Role: LEADER
I20260809 03:34:17.800222 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:17.800479 16004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260809 03:34:17.801020 15999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:17.801460 16004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13955c214ab2450484031930e729b117. Latest consensus state: current_term: 1 leader_uuid: "13955c214ab2450484031930e729b117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260809 03:34:17.801573 16004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:17.801909 16005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13955c214ab2450484031930e729b117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13955c214ab2450484031930e729b117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260809 03:34:17.802000 16005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:17.802045 16021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:17.805130 16021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:17.810738 16021 catalog_manager.cc:1347] Generated new cluster ID: f454b87ac0214c78947bb92d64ee5e6e
I20260809 03:34:17.810835 16021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:17.812510 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:17.803344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52547 (local address 127.0.0.1:37209)
0809 03:34:17.803809 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:17.803813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:17.804018 (+ 205us) server_negotiation.cc:407] Connection header received
0809 03:34:17.804205 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:17.804208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:17.804268 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:17.804364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:17.805167 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:17.805901 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:17.807127 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:17.807475 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:17.807589 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:17.809134 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:17.809138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:17.809141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:17.809167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:17.810687 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:17.811267 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:17.811271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:17.811272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:17.811367 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:17.811678 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:17.811681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:17.811683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:17.811894 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:17.812016 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:17.812245 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 03:34:17.812351 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260809 03:34:17.826409 16021 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:17.827291 16021 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:17.833384 16021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13955c214ab2450484031930e729b117: Generated new TSK 0
I20260809 03:34:17.834048 16021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:17.839591 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 15897
2026-08-09T03:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[16047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[16047](info): set up 2 sockets
Aug 09 03:34:17 dist-test-slave-cvwc krb5kdc[16047](info): commencing operation
krb5kdc: starting...
W20260809 03:34:18.989095 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:19 dist-test-slave-cvwc krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:19.106240 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:19.216277 16062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:19.216526 16062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:19.216565 16062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:19.220461 16062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:19.220515 16062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:19.220533 16062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:19.220548 16062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:19.220561 16062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:19.225665 16062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:19.226627 16062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:19.227576 16062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:19.235817 16072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:19.235864 16069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:19.235862 16062 server_base.cc:1034] running on GCE node
W20260809 03:34:19.235813 16070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:19.236543 16062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:19.237541 16062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:19.238694 16062 hybrid_clock.cc:648] HybridClock initialized: now 1786246459238690 us; error 50 us; skew 500 ppm
Aug 09 03:34:19 dist-test-slave-cvwc krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:19.241215 16062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:19.242051 16062 webserver.cc:460] Webserver started at http://127.0.0.1:34113/ using document root <none> and password file <none>
I20260809 03:34:19.242543 16062 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:19.242597 16062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:19.242758 16062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:19.244640 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "bdd1a7fbcd47430a89b460ee098b77ca"
format_stamp: "Formatted at 2026-08-09 03:34:19 on dist-test-slave-cvwc"
server_key: "31455f504bfb1e3f45a7ae3a53c04316"
server_key_iv: "640c9a084cacdc1f581d9356eae96dab"
server_key_version: "encryptionkey@0"
I20260809 03:34:19.245131 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "bdd1a7fbcd47430a89b460ee098b77ca"
format_stamp: "Formatted at 2026-08-09 03:34:19 on dist-test-slave-cvwc"
server_key: "31455f504bfb1e3f45a7ae3a53c04316"
server_key_iv: "640c9a084cacdc1f581d9356eae96dab"
server_key_version: "encryptionkey@0"
I20260809 03:34:19.248564 16062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:19.250756 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:19.251838 16062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:19.251941 16062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "bdd1a7fbcd47430a89b460ee098b77ca"
format_stamp: "Formatted at 2026-08-09 03:34:19 on dist-test-slave-cvwc"
server_key: "31455f504bfb1e3f45a7ae3a53c04316"
server_key_iv: "640c9a084cacdc1f581d9356eae96dab"
server_key_version: "encryptionkey@0"
I20260809 03:34:19.252022 16062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:19.279335 16062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:19.282047 16062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:19.282202 16062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:19.288841 16062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33437
I20260809 03:34:19.288853 16164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33437 every 8 connection(s)
I20260809 03:34:19.289763 16062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:19.292040 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16062
I20260809 03:34:19.292191 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:19.292500 16249 external_mini_cluster.cc:1428] Setting key 1b6f757a61d134156f8d841079ea693c
I20260809 03:34:19.292665 16166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:19.299325 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca: Bootstrap starting.
I20260809 03:34:19.301575 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:19.293902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60653 (local address 127.0.0.1:33437)
0809 03:34:19.294406 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:19.294422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:19.294467 (+ 45us) server_negotiation.cc:407] Connection header received
0809 03:34:19.295390 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:19.295410 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:19.295778 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:19.296064 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:19.296377 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:19.297420 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:19.298642 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:19.299100 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:19.299213 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:19.300009 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:19.300013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:19.300024 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:19.300370 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:19.300424 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:19.300447 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:19.300518 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:19.300663 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 03:34:19.300857 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":155,"threads_started":1}
I20260809 03:34:19.301636 16166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:19.302343 16166 log.cc:824] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:19.304179 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:19.304263 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca: No bootstrap required, opened a new log
I20260809 03:34:19.308424 16166 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260809 03:34:19.308688 16166 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:19.308792 16166 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd1a7fbcd47430a89b460ee098b77ca, State: Initialized, Role: FOLLOWER
I20260809 03:34:19.309235 16166 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260809 03:34:19.309358 16166 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:19.309415 16166 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:19.309509 16166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:19.309908 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:19.306278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54521 (local address 127.0.0.1:33437)
0809 03:34:19.306380 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:19.306383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:19.306496 (+ 113us) server_negotiation.cc:407] Connection header received
0809 03:34:19.306572 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:19.306574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:19.306614 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:19.306693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:19.306877 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:19.307623 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:19.308346 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:19.308672 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:19.308776 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:19.309486 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:19.309489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:19.309492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:19.309504 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:19.309531 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:19.309544 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:19.309617 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:19.309744 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 03:34:19.309794 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:19.310483 16166 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260809 03:34:19.310789 16166 leader_election.cc:302] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdd1a7fbcd47430a89b460ee098b77ca; no voters:
I20260809 03:34:19.311043 16166 leader_election.cc:288] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:19.311115 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:19.311323 16178 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [term 1 LEADER]: Becoming Leader. State: Replica: bdd1a7fbcd47430a89b460ee098b77ca, State: Running, Role: LEADER
I20260809 03:34:19.311636 16178 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } }
I20260809 03:34:19.312140 16166 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:19.313330 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } } }
I20260809 03:34:19.313346 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd1a7fbcd47430a89b460ee098b77ca. Latest consensus state: current_term: 1 leader_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd1a7fbcd47430a89b460ee098b77ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33437 } } }
I20260809 03:34:19.313534 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:19.313555 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:19.313978 16191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:19.316941 16191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:19.322352 16191 catalog_manager.cc:1347] Generated new cluster ID: a5dd8327977848c0af221efa72d5d1dd
I20260809 03:34:19.322417 16191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:19.340538 16191 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:19.341778 16191 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:19.347321 16191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdd1a7fbcd47430a89b460ee098b77ca: Generated new TSK 0
I20260809 03:34:19.348059 16191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:19.356832 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16062
2026-08-09T03:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:19 dist-test-slave-cvwc krb5kdc[16209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:19 dist-test-slave-cvwc krb5kdc[16209](info): set up 2 sockets
Aug 09 03:34:19 dist-test-slave-cvwc krb5kdc[16209](info): commencing operation
krb5kdc: starting...
W20260809 03:34:20.497342 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:20 dist-test-slave-cvwc krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:20.610330 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:20.718428 16230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:20.718695 16230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:20.718755 16230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:20.722648 16230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:20.722699 16230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:20.722724 16230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:20.722747 16230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:20.722775 16230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:20.727737 16230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:20.728657 16230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:20.729604 16230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:20.737484 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:20.737495 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:20.737669 16230 server_base.cc:1034] running on GCE node
W20260809 03:34:20.737500 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:20.738312 16230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:20.739319 16230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:20.740474 16230 hybrid_clock.cc:648] HybridClock initialized: now 1786246460740462 us; error 22 us; skew 500 ppm
Aug 09 03:34:20 dist-test-slave-cvwc krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:20.743064 16230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:20.743878 16230 webserver.cc:460] Webserver started at http://127.0.0.1:44485/ using document root <none> and password file <none>
I20260809 03:34:20.744376 16230 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:20.744448 16230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:20.744638 16230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:20.746639 16230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "ba96cbd4d50542c089b676c5743e2320"
format_stamp: "Formatted at 2026-08-09 03:34:20 on dist-test-slave-cvwc"
server_key: "f7576d4564e7ad7ed7e52915a8224b1a"
server_key_iv: "14d50af74ada984278ae82ba342c350a"
server_key_version: "encryptionkey@0"
I20260809 03:34:20.747229 16230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "ba96cbd4d50542c089b676c5743e2320"
format_stamp: "Formatted at 2026-08-09 03:34:20 on dist-test-slave-cvwc"
server_key: "f7576d4564e7ad7ed7e52915a8224b1a"
server_key_iv: "14d50af74ada984278ae82ba342c350a"
server_key_version: "encryptionkey@0"
I20260809 03:34:20.750664 16230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:20.752892 16260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:20.754063 16230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:20.754173 16230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "ba96cbd4d50542c089b676c5743e2320"
format_stamp: "Formatted at 2026-08-09 03:34:20 on dist-test-slave-cvwc"
server_key: "f7576d4564e7ad7ed7e52915a8224b1a"
server_key_iv: "14d50af74ada984278ae82ba342c350a"
server_key_version: "encryptionkey@0"
I20260809 03:34:20.754262 16230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:20.758644 16230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:20.761528 16230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:20.761704 16230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:20.768118 16230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44613
I20260809 03:34:20.768147 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44613 every 8 connection(s)
I20260809 03:34:20.769004 16230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:20.771631 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:20.775329 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16230
I20260809 03:34:20.775449 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:20.775698 16249 external_mini_cluster.cc:1428] Setting key dd7d476f4ecd8754fdcf033f82086130
I20260809 03:34:20.777243 16347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320: Bootstrap starting.
I20260809 03:34:20.779500 16347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:20.780268 16347 log.cc:824] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:20.780681 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:20.777010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42613 (local address 127.0.0.1:44613)
0809 03:34:20.777444 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:20.777459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:20.777495 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:34:20.778158 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:20.778177 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:20.778448 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:20.778771 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:20.779519 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:20.779524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:20.779536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:20.779547 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:20.779615 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:20.779641 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:20.779734 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:20.779846 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 03:34:20.780005 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260809 03:34:20.782176 16347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320: No bootstrap required, opened a new log
I20260809 03:34:20.783373 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:20.785912 16347 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } }
I20260809 03:34:20.786176 16347 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:20.786239 16347 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba96cbd4d50542c089b676c5743e2320, State: Initialized, Role: FOLLOWER
I20260809 03:34:20.786780 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:20.785237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44271 (local address 127.0.0.1:44613)
0809 03:34:20.785329 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:20.785333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:20.785523 (+ 190us) server_negotiation.cc:407] Connection header received
0809 03:34:20.785621 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:20.785624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:20.785653 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:20.785730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:20.786371 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:20.786375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:20.786377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:20.786397 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:20.786426 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:20.786440 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:20.786528 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:20.786605 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 03:34:20.786639 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:34:20.786691 16347 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } }
I20260809 03:34:20.786878 16347 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:20.786972 16347 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:20.787045 16347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:20.788019 16347 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } }
I20260809 03:34:20.788303 16347 leader_election.cc:302] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba96cbd4d50542c089b676c5743e2320; no voters:
I20260809 03:34:20.788529 16347 leader_election.cc:288] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:20.788618 16362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:20.788810 16362 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [term 1 LEADER]: Becoming Leader. State: Replica: ba96cbd4d50542c089b676c5743e2320, State: Running, Role: LEADER
I20260809 03:34:20.789143 16362 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } }
I20260809 03:34:20.789613 16347 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:20.790723 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba96cbd4d50542c089b676c5743e2320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } } }
I20260809 03:34:20.790845 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:20.791211 16368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:20.792291 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba96cbd4d50542c089b676c5743e2320. Latest consensus state: current_term: 1 leader_uuid: "ba96cbd4d50542c089b676c5743e2320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba96cbd4d50542c089b676c5743e2320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44613 } } }
I20260809 03:34:20.792392 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:20.794646 16368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:20.800153 16368 catalog_manager.cc:1347] Generated new cluster ID: 935f936f24da4b4c8d373eb84be12d29
I20260809 03:34:20.800211 16368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:20.817761 16368 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:20.818068 16368 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:20.840512 16368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba96cbd4d50542c089b676c5743e2320: Generated new TSK 0
I20260809 03:34:20.841126 16368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:20.851468 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16230
2026-08-09T03:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:20 dist-test-slave-cvwc krb5kdc[16394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:20 dist-test-slave-cvwc krb5kdc[16394](info): set up 2 sockets
Aug 09 03:34:20 dist-test-slave-cvwc krb5kdc[16394](info): commencing operation
krb5kdc: starting...
W20260809 03:34:22.001356 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:22.118024 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:22.226234 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:22.226490 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:22.226581 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:22.230582 16409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:22.230639 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:22.230656 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:22.230674 16409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:22.230690 16409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:22.235729 16409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:22.236680 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:22.237638 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:22.245493 16419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:22.245514 16421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:22.245527 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:22.245800 16409 server_base.cc:1034] running on GCE node
I20260809 03:34:22.246326 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:22.247315 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:22.248463 16409 hybrid_clock.cc:648] HybridClock initialized: now 1786246462248440 us; error 34 us; skew 500 ppm
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:22.250977 16409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:22.251791 16409 webserver.cc:460] Webserver started at http://127.0.0.1:37489/ using document root <none> and password file <none>
I20260809 03:34:22.252291 16409 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:22.252357 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:22.252566 16409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:22.254578 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "a9995d4a2e8d47deab72b7887b2319b0"
format_stamp: "Formatted at 2026-08-09 03:34:22 on dist-test-slave-cvwc"
server_key: "ed473c1510558316c0598043fb71f5f2"
server_key_iv: "df4c9bf7bedfcfd16b879f3de0e0e1fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:22.255101 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "a9995d4a2e8d47deab72b7887b2319b0"
format_stamp: "Formatted at 2026-08-09 03:34:22 on dist-test-slave-cvwc"
server_key: "ed473c1510558316c0598043fb71f5f2"
server_key_iv: "df4c9bf7bedfcfd16b879f3de0e0e1fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:22.258522 16409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:22.260639 16441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:22.261792 16409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 03:34:22.261901 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "a9995d4a2e8d47deab72b7887b2319b0"
format_stamp: "Formatted at 2026-08-09 03:34:22 on dist-test-slave-cvwc"
server_key: "ed473c1510558316c0598043fb71f5f2"
server_key_iv: "df4c9bf7bedfcfd16b879f3de0e0e1fd"
server_key_version: "encryptionkey@0"
I20260809 03:34:22.261987 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:22.276475 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:22.278910 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:22.279078 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:22.285656 16409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39655
I20260809 03:34:22.285671 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39655 every 8 connection(s)
I20260809 03:34:22.286571 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:22.289140 16519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:22.293066 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16409
I20260809 03:34:22.293200 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:22.293469 16249 external_mini_cluster.cc:1428] Setting key c76d163f3a7fa93cea73aa69d15bdfd8
I20260809 03:34:22.294809 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0: Bootstrap starting.
I20260809 03:34:22.297106 16519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:22.297887 16519 log.cc:824] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:22.299554 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0: No bootstrap required, opened a new log
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:34:22.303052 16519 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260809 03:34:22.303304 16519 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:22.303364 16519 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9995d4a2e8d47deab72b7887b2319b0, State: Initialized, Role: FOLLOWER
I20260809 03:34:22.303745 16519 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260809 03:34:22.303870 16519 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:22.303934 16519 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:22.304033 16519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:22.304876 16519 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260809 03:34:22.305176 16519 leader_election.cc:302] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9995d4a2e8d47deab72b7887b2319b0; no voters:
I20260809 03:34:22.305445 16519 leader_election.cc:288] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:22.305599 16527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:22.305874 16527 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [term 1 LEADER]: Becoming Leader. State: Replica: a9995d4a2e8d47deab72b7887b2319b0, State: Running, Role: LEADER
I20260809 03:34:22.306200 16527 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260809 03:34:22.306658 16519 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:22.306887 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:22.294823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55015 (local address 127.0.0.1:39655)
0809 03:34:22.295209 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:22.295227 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:22.295262 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:34:22.295866 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:22.295882 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:22.296204 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:22.296460 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:22.297346 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:22.298375 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:22.299230 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:22.299696 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:22.299835 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:22.302222 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:22.302228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:22.302240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:22.302267 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:22.303892 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:22.304406 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:22.304412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:22.304418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:22.304524 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:22.304896 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:22.304899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:22.304901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:22.305292 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:22.305508 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:22.305801 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 03:34:22.306028 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260809 03:34:22.307462 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9995d4a2e8d47deab72b7887b2319b0. Latest consensus state: current_term: 1 leader_uuid: "a9995d4a2e8d47deab72b7887b2319b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } } }
I20260809 03:34:22.307583 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:22.307833 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9995d4a2e8d47deab72b7887b2319b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9995d4a2e8d47deab72b7887b2319b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } } }
I20260809 03:34:22.307932 16542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:22.307922 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:22.310206 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:22.311712 16542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:22.317051 16542 catalog_manager.cc:1347] Generated new cluster ID: d3343911d2ff4c81a374e2f8cc2b9e37
I20260809 03:34:22.317111 16542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:22.321856 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:22.312303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:33035 (local address 127.0.0.1:39655)
0809 03:34:22.312414 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:22.312417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:22.313615 (+ 1198us) server_negotiation.cc:407] Connection header received
0809 03:34:22.313737 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:22.313740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:22.313787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:22.313850 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:22.314450 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:22.314935 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:22.315887 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:22.316237 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:22.316340 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:22.318658 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:22.318662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:22.318665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:22.318690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:22.320205 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:22.320674 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:22.320678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:22.320680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:22.320764 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:22.321044 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:22.321047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:22.321048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:22.321249 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:22.321384 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:22.321600 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 03:34:22.321690 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:22.325976 16542 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:22.327036 16542 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:22.334501 16542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9995d4a2e8d47deab72b7887b2319b0: Generated new TSK 0
I20260809 03:34:22.335065 16542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:22.340966 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16409
W20260809 03:34:22.360991 16392 connection.cc:535] client connection to 127.0.0.1:39655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39655): Connection reset by peer (error 104)
2026-08-09T03:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16566](info): set up 2 sockets
Aug 09 03:34:22 dist-test-slave-cvwc krb5kdc[16566](info): commencing operation
krb5kdc: starting...
W20260809 03:34:23.494551 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:23 dist-test-slave-cvwc krb5kdc[16566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:23.608119 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:23.716799 16588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:23.717036 16588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:23.717074 16588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:23.721167 16588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:23.721228 16588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:23.721248 16588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:23.721279 16588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:23.721308 16588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:23.726538 16588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:23.727471 16588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:23.728413 16588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:23.736270 16594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:23.736301 16595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:23.736406 16588 server_base.cc:1034] running on GCE node
W20260809 03:34:23.736331 16598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:23.737114 16588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:23.738173 16588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:23.739331 16588 hybrid_clock.cc:648] HybridClock initialized: now 1786246463739318 us; error 26 us; skew 500 ppm
Aug 09 03:34:23 dist-test-slave-cvwc krb5kdc[16566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:23.744985 16588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:23.745797 16588 webserver.cc:460] Webserver started at http://127.0.0.1:36191/ using document root <none> and password file <none>
I20260809 03:34:23.746299 16588 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:23.746354 16588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:23.746519 16588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:23.748414 16588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "c2f7120101d54c5ebe47173c1de00c2b"
format_stamp: "Formatted at 2026-08-09 03:34:23 on dist-test-slave-cvwc"
server_key: "f2134a944eae05c4820ac817f2eea358"
server_key_iv: "5da6ea769bb3e79f0590dbc52ab9613d"
server_key_version: "encryptionkey@0"
I20260809 03:34:23.748917 16588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "c2f7120101d54c5ebe47173c1de00c2b"
format_stamp: "Formatted at 2026-08-09 03:34:23 on dist-test-slave-cvwc"
server_key: "f2134a944eae05c4820ac817f2eea358"
server_key_iv: "5da6ea769bb3e79f0590dbc52ab9613d"
server_key_version: "encryptionkey@0"
I20260809 03:34:23.752435 16588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:34:23.754729 16611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:23.755860 16588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:23.755976 16588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "c2f7120101d54c5ebe47173c1de00c2b"
format_stamp: "Formatted at 2026-08-09 03:34:23 on dist-test-slave-cvwc"
server_key: "f2134a944eae05c4820ac817f2eea358"
server_key_iv: "5da6ea769bb3e79f0590dbc52ab9613d"
server_key_version: "encryptionkey@0"
I20260809 03:34:23.756057 16588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:23.780421 16588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:23.783154 16588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:23.783304 16588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:23.790017 16588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42397
I20260809 03:34:23.790048 16698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42397 every 8 connection(s)
I20260809 03:34:23.790917 16588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:23.793561 16701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:23.793790 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16588
I20260809 03:34:23.793922 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:23.794179 16249 external_mini_cluster.cc:1428] Setting key d83960be64842feea820e23dd8c48972
I20260809 03:34:23.799571 16701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: Bootstrap starting.
I20260809 03:34:23.801729 16701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:23.802456 16701 log.cc:824] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:23.802536 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:23.795463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:53523 (local address 127.0.0.1:42397)
0809 03:34:23.795888 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:23.795907 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:23.795941 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:34:23.796619 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:23.796634 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:23.796950 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:23.797243 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:23.797648 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:23.798644 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:23.799491 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:23.799971 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:23.800113 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:23.800957 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:23.800962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:23.800973 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:23.801318 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:23.801369 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:23.801400 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:23.801512 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:23.801699 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 03:34:23.801865 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":143,"threads_started":1}
I20260809 03:34:23.804379 16701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: No bootstrap required, opened a new log
I20260809 03:34:23.805253 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:23.808061 16701 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260809 03:34:23.808271 16701 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:23.808338 16701 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f7120101d54c5ebe47173c1de00c2b, State: Initialized, Role: FOLLOWER
I20260809 03:34:23.808799 16701 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260809 03:34:23.808952 16701 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:23.809010 16701 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:23.809098 16701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:23.810051 16701 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260809 03:34:23.810407 16701 leader_election.cc:302] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f7120101d54c5ebe47173c1de00c2b; no voters:
I20260809 03:34:23.810717 16701 leader_election.cc:288] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:23.810845 16719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:23.810860 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:23.807106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:45707 (local address 127.0.0.1:42397)
0809 03:34:23.807204 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:23.807208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:23.807356 (+ 148us) server_negotiation.cc:407] Connection header received
0809 03:34:23.807466 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:23.807468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:23.807506 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:23.807563 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:23.807817 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:23.808291 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:23.809338 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:23.809576 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:23.809649 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:23.810434 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:23.810437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:23.810439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:23.810449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:23.810472 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:23.810481 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:23.810561 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:23.810708 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 03:34:23.810740 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:34:23.811430 16719 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [term 1 LEADER]: Becoming Leader. State: Replica: c2f7120101d54c5ebe47173c1de00c2b, State: Running, Role: LEADER
I20260809 03:34:23.811780 16719 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260809 03:34:23.811957 16701 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:23.813329 16722 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f7120101d54c5ebe47173c1de00c2b. Latest consensus state: current_term: 1 leader_uuid: "c2f7120101d54c5ebe47173c1de00c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } } }
I20260809 03:34:23.813445 16722 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:23.813732 16721 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f7120101d54c5ebe47173c1de00c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f7120101d54c5ebe47173c1de00c2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } } }
I20260809 03:34:23.813818 16721 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:23.816190 16739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:23.816255 16739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:23.816581 16735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:23.817232 16735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:23.822469 16735 catalog_manager.cc:1347] Generated new cluster ID: fe81bd242d704d33a74869effff9b531
I20260809 03:34:23.822530 16735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:23.836544 16735 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:23.837363 16735 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:23.840942 16735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f7120101d54c5ebe47173c1de00c2b: Generated new TSK 0
I20260809 03:34:23.841513 16735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:23.850265 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16588
W20260809 03:34:23.867463 16564 connection.cc:535] client connection to 127.0.0.1:42397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42397): Connection reset by peer (error 104)
2026-08-09T03:34:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:23 dist-test-slave-cvwc krb5kdc[16747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:23 dist-test-slave-cvwc krb5kdc[16747](info): set up 2 sockets
Aug 09 03:34:23 dist-test-slave-cvwc krb5kdc[16747](info): commencing operation
krb5kdc: starting...
W20260809 03:34:25.014132 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:25 dist-test-slave-cvwc krb5kdc[16747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:25.129961 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:25.237847 16767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:25.238085 16767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:25.238124 16767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:25.241993 16767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:25.242043 16767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:25.242066 16767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:25.242095 16767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:25.242172 16767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:25.247241 16767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:25.248129 16767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:25.249058 16767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:25.257565 16777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:25.257628 16782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:25.257730 16767 server_base.cc:1034] running on GCE node
W20260809 03:34:25.257565 16778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:25.258412 16767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:25.259518 16767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:25.260674 16767 hybrid_clock.cc:648] HybridClock initialized: now 1786246465260655 us; error 31 us; skew 500 ppm
Aug 09 03:34:25 dist-test-slave-cvwc krb5kdc[16747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:25.263275 16767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:25.264134 16767 webserver.cc:460] Webserver started at http://127.0.0.1:40779/ using document root <none> and password file <none>
I20260809 03:34:25.264678 16767 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:25.264750 16767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:25.264962 16767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:25.266966 16767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "8f5190dee7fe49a399a55c25ec881201"
format_stamp: "Formatted at 2026-08-09 03:34:25 on dist-test-slave-cvwc"
server_key: "09d3b9a02664f8be6181ea89fec7048c"
server_key_iv: "0ceb34e09235c261d535b536c563d45d"
server_key_version: "encryptionkey@0"
I20260809 03:34:25.267498 16767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "8f5190dee7fe49a399a55c25ec881201"
format_stamp: "Formatted at 2026-08-09 03:34:25 on dist-test-slave-cvwc"
server_key: "09d3b9a02664f8be6181ea89fec7048c"
server_key_iv: "0ceb34e09235c261d535b536c563d45d"
server_key_version: "encryptionkey@0"
I20260809 03:34:25.271013 16767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:25.273222 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:25.274359 16767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:25.274472 16767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "8f5190dee7fe49a399a55c25ec881201"
format_stamp: "Formatted at 2026-08-09 03:34:25 on dist-test-slave-cvwc"
server_key: "09d3b9a02664f8be6181ea89fec7048c"
server_key_iv: "0ceb34e09235c261d535b536c563d45d"
server_key_version: "encryptionkey@0"
I20260809 03:34:25.274560 16767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:25.278844 16767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:25.281746 16767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:25.281909 16767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:25.288412 16767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36189
I20260809 03:34:25.288662 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36189 every 8 connection(s)
I20260809 03:34:25.289270 16767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:25.291944 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:25.295221 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16767
I20260809 03:34:25.295344 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:25.295608 16249 external_mini_cluster.cc:1428] Setting key 23f9938a0c4ed2944babc0a3d4ed2ea6
I20260809 03:34:25.297744 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201: Bootstrap starting.
I20260809 03:34:25.300449 16882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:25.300904 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:25.296860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:54047 (local address 127.0.0.1:36189)
0809 03:34:25.297338 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:25.297356 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:25.297397 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:34:25.298221 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:25.298245 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:25.298598 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:25.298970 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:25.299584 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:25.299589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:25.299603 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:25.299612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:25.299661 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:25.299687 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:25.299806 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:25.299880 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 03:34:25.300025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":127,"threads_started":1}
I20260809 03:34:25.301308 16882 log.cc:824] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:25.303169 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201: No bootstrap required, opened a new log
I20260809 03:34:25.303493 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:25.307153 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:25.305792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:33421 (local address 127.0.0.1:36189)
0809 03:34:25.305902 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:25.305906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:25.306030 (+ 124us) server_negotiation.cc:407] Connection header received
0809 03:34:25.306114 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:25.306117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:25.306150 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:25.306234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:25.306812 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:25.306815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:25.306818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:25.306828 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:25.306854 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:25.306866 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:25.306952 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:25.306997 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 03:34:25.307034 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:25.307639 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260809 03:34:25.307925 16882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:25.307994 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5190dee7fe49a399a55c25ec881201, State: Initialized, Role: FOLLOWER
I20260809 03:34:25.308454 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260809 03:34:25.308581 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:25.308645 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:25.308746 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:25.309645 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260809 03:34:25.309993 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5190dee7fe49a399a55c25ec881201; no voters:
I20260809 03:34:25.310307 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:25.310382 16897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:25.310602 16897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5190dee7fe49a399a55c25ec881201, State: Running, Role: LEADER
I20260809 03:34:25.310886 16897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260809 03:34:25.311494 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:25.312424 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5190dee7fe49a399a55c25ec881201. Latest consensus state: current_term: 1 leader_uuid: "8f5190dee7fe49a399a55c25ec881201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } } }
I20260809 03:34:25.312547 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:25.313649 16910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:25.313633 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5190dee7fe49a399a55c25ec881201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5190dee7fe49a399a55c25ec881201" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } } }
I20260809 03:34:25.313879 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:25.316350 16910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:25.321764 16910 catalog_manager.cc:1347] Generated new cluster ID: 9713368df4ba480782c2d9a54c7be26c
I20260809 03:34:25.321832 16910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:25.352221 16910 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:25.352583 16910 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:25.365504 16910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5190dee7fe49a399a55c25ec881201: Generated new TSK 0
I20260809 03:34:25.366392 16910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:25.380555 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16767
2026-08-09T03:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:25 dist-test-slave-cvwc krb5kdc[16928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:25 dist-test-slave-cvwc krb5kdc[16928](info): set up 2 sockets
Aug 09 03:34:25 dist-test-slave-cvwc krb5kdc[16928](info): commencing operation
krb5kdc: starting...
W20260809 03:34:26.516688 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:26.629676 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:26.739110 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:26.739352 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:26.739393 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:26.743280 16948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:26.743332 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:26.743353 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:26.743383 16948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:26.743409 16948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:26.748469 16948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:26.749426 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:26.750423 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:26.758530 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:26.758543 16948 server_base.cc:1034] running on GCE node
W20260809 03:34:26.758549 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:26.758562 16961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:26.759305 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:26.760318 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:26.761482 16948 hybrid_clock.cc:648] HybridClock initialized: now 1786246466761460 us; error 33 us; skew 500 ppm
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:26.764293 16948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:26.765177 16948 webserver.cc:460] Webserver started at http://127.0.0.1:37867/ using document root <none> and password file <none>
I20260809 03:34:26.765702 16948 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:26.765802 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:26.766021 16948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:26.767998 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd"
format_stamp: "Formatted at 2026-08-09 03:34:26 on dist-test-slave-cvwc"
server_key: "ebbd4a0038ee4ce372668d2cc05aed95"
server_key_iv: "f7d2f5778d663703831e3e9a846071a9"
server_key_version: "encryptionkey@0"
I20260809 03:34:26.768527 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd"
format_stamp: "Formatted at 2026-08-09 03:34:26 on dist-test-slave-cvwc"
server_key: "ebbd4a0038ee4ce372668d2cc05aed95"
server_key_iv: "f7d2f5778d663703831e3e9a846071a9"
server_key_version: "encryptionkey@0"
I20260809 03:34:26.772112 16948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 03:34:26.774407 16971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:26.775628 16948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:26.775738 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd"
format_stamp: "Formatted at 2026-08-09 03:34:26 on dist-test-slave-cvwc"
server_key: "ebbd4a0038ee4ce372668d2cc05aed95"
server_key_iv: "f7d2f5778d663703831e3e9a846071a9"
server_key_version: "encryptionkey@0"
I20260809 03:34:26.775835 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:26.795787 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:26.798521 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:26.798695 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:26.805490 16948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34443
I20260809 03:34:26.805518 17051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34443 every 8 connection(s)
I20260809 03:34:26.806455 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:26.809176 17053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:26.814848 17053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: Bootstrap starting.
I20260809 03:34:26.814868 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 16948
I20260809 03:34:26.814971 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:26.815222 16249 external_mini_cluster.cc:1428] Setting key c197602a12c466c9584ca706ea70c7bf
I20260809 03:34:26.817528 17053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:26.818502 17053 log.cc:824] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:26.820822 17053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: No bootstrap required, opened a new log
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 03:34:26.825011 17053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } }
I20260809 03:34:26.825302 17053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:26.825369 17053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d2fbfe0c664b6fadd5393f36ee8cdd, State: Initialized, Role: FOLLOWER
I20260809 03:34:26.825870 17053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } }
I20260809 03:34:26.826006 17053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:26.826074 17053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:26.826162 17053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:26.826984 17053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } }
I20260809 03:34:26.827370 17053 leader_election.cc:302] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d2fbfe0c664b6fadd5393f36ee8cdd; no voters:
I20260809 03:34:26.827742 17053 leader_election.cc:288] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:26.827821 17062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:26.828020 17062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [term 1 LEADER]: Becoming Leader. State: Replica: 35d2fbfe0c664b6fadd5393f36ee8cdd, State: Running, Role: LEADER
I20260809 03:34:26.828347 17062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } }
I20260809 03:34:26.829066 17053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:26.829530 17065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } } }
I20260809 03:34:26.829542 17062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d2fbfe0c664b6fadd5393f36ee8cdd. Latest consensus state: current_term: 1 leader_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d2fbfe0c664b6fadd5393f36ee8cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34443 } } }
I20260809 03:34:26.829777 17065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:26.829800 17062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:26.830147 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:26.816485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:38921 (local address 127.0.0.1:34443)
0809 03:34:26.816905 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:26.816925 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:26.816969 (+ 44us) server_negotiation.cc:407] Connection header received
0809 03:34:26.817852 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:26.817879 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:26.818324 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:26.818713 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:26.819494 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:26.820916 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:26.821771 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:26.822460 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:26.822643 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:26.825254 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:26.825259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:26.825273 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:26.825305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:26.826923 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:26.827547 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:26.827553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:26.827559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:26.827670 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:26.828027 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:26.828030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:26.828032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:26.828451 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:26.828654 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:26.828980 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 03:34:26.829202 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260809 03:34:26.833611 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260809 03:34:26.834004 17089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:26.834081 17089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:26.834158 17070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:26.834980 17070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:26.840636 17070 catalog_manager.cc:1347] Generated new cluster ID: 87471d0c4767496d9337fb998b6fec1b
I20260809 03:34:26.840752 17070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:26.844214 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:26.835620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:48237 (local address 127.0.0.1:34443)
0809 03:34:26.835739 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:26.835743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:26.835974 (+ 231us) server_negotiation.cc:407] Connection header received
0809 03:34:26.836081 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:26.836085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:26.836128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:26.836196 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:26.836892 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:26.837638 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:26.838828 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:26.839095 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:26.839209 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:26.840749 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:26.840752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:26.840755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:26.840780 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:26.842419 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:26.842983 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:26.842987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:26.842989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:26.843075 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:26.843389 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:26.843392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:26.843394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:26.843592 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:26.843733 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:26.843988 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 03:34:26.844079 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:34:26.849023 17070 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:26.850207 17070 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:26.865089 17070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d2fbfe0c664b6fadd5393f36ee8cdd: Generated new TSK 0
I20260809 03:34:26.865947 17070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:26.877481 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 16948
W20260809 03:34:26.892760 16926 connection.cc:535] client connection to 127.0.0.1:34443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34443): Connection reset by peer (error 104)
2026-08-09T03:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[17105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[17105](info): set up 2 sockets
Aug 09 03:34:26 dist-test-slave-cvwc krb5kdc[17105](info): commencing operation
krb5kdc: starting...
W20260809 03:34:28.016551 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:28 dist-test-slave-cvwc krb5kdc[17105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:28.131598 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:28.239244 17123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:28.239506 17123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:28.239550 17123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:28.243495 17123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:28.243548 17123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:28.243567 17123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:28.243597 17123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:28.243623 17123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:28.248901 17123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:28.249962 17123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:28.250993 17123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:28.259245 17130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:28.259254 17131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:28.259255 17133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:28.259579 17123 server_base.cc:1034] running on GCE node
I20260809 03:34:28.260099 17123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:28.261118 17123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:28.262259 17123 hybrid_clock.cc:648] HybridClock initialized: now 1786246468262233 us; error 38 us; skew 500 ppm
Aug 09 03:34:28 dist-test-slave-cvwc krb5kdc[17105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:28.264853 17123 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:28.265690 17123 webserver.cc:460] Webserver started at http://127.0.0.1:45375/ using document root <none> and password file <none>
I20260809 03:34:28.266261 17123 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:28.266319 17123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:28.266525 17123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:28.268499 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "3f3eec2bbf334054a9fad592874ef510"
format_stamp: "Formatted at 2026-08-09 03:34:28 on dist-test-slave-cvwc"
server_key: "abd3989a4520c30cbd6d513cbdcefe63"
server_key_iv: "84eb7b3d06689bda3bd64f05cf11bd08"
server_key_version: "encryptionkey@0"
I20260809 03:34:28.269022 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "3f3eec2bbf334054a9fad592874ef510"
format_stamp: "Formatted at 2026-08-09 03:34:28 on dist-test-slave-cvwc"
server_key: "abd3989a4520c30cbd6d513cbdcefe63"
server_key_iv: "84eb7b3d06689bda3bd64f05cf11bd08"
server_key_version: "encryptionkey@0"
I20260809 03:34:28.272593 17123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:28.274850 17142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:28.275987 17123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:28.276099 17123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "3f3eec2bbf334054a9fad592874ef510"
format_stamp: "Formatted at 2026-08-09 03:34:28 on dist-test-slave-cvwc"
server_key: "abd3989a4520c30cbd6d513cbdcefe63"
server_key_iv: "84eb7b3d06689bda3bd64f05cf11bd08"
server_key_version: "encryptionkey@0"
I20260809 03:34:28.276190 17123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:28.292934 17123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:28.295876 17123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:28.296047 17123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:28.302659 17123 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36113
I20260809 03:34:28.302677 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36113 every 8 connection(s)
I20260809 03:34:28.303619 17123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:28.306232 17228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:28.306694 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17123
I20260809 03:34:28.306823 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:28.307126 16249 external_mini_cluster.cc:1428] Setting key 81f9b2b06f0ae92697477b1697e4d449
I20260809 03:34:28.312536 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510: Bootstrap starting.
I20260809 03:34:28.314632 17228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:28.315347 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:28.308456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:57979 (local address 127.0.0.1:36113)
0809 03:34:28.308859 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:28.308873 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:28.308909 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:34:28.309570 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:28.309586 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:28.309950 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:28.310244 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:28.310624 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:28.311550 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:28.312403 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:28.312878 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:28.313026 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:28.313833 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:28.313838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:28.313849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:28.314195 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:28.314245 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:28.314268 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:28.314378 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:28.314553 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 03:34:28.314674 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260809 03:34:28.315335 17228 log.cc:824] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:28.317214 17228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510: No bootstrap required, opened a new log
I20260809 03:34:28.317863 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:28.321316 17228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260809 03:34:28.321576 17228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:28.321635 17228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3eec2bbf334054a9fad592874ef510, State: Initialized, Role: FOLLOWER
I20260809 03:34:28.322096 17228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260809 03:34:28.322257 17228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:28.322325 17228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:28.322419 17228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:28.323427 17228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260809 03:34:28.323734 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:28.319988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:58799 (local address 127.0.0.1:36113)
0809 03:34:28.320087 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:28.320091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:28.320240 (+ 149us) server_negotiation.cc:407] Connection header received
0809 03:34:28.320348 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:28.320350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:28.320389 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:28.320448 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:28.320696 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:28.321158 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:28.322220 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:28.322546 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:28.322625 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:28.323315 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:28.323318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:28.323320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:28.323330 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:28.323352 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:28.323361 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:28.323431 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:28.323586 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 03:34:28.323616 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:34:28.323776 17228 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3eec2bbf334054a9fad592874ef510; no voters:
I20260809 03:34:28.324079 17228 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:28.324159 17242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:28.324327 17242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3eec2bbf334054a9fad592874ef510, State: Running, Role: LEADER
I20260809 03:34:28.324682 17242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } }
I20260809 03:34:28.325269 17228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:28.325851 17242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3eec2bbf334054a9fad592874ef510. Latest consensus state: current_term: 1 leader_uuid: "3f3eec2bbf334054a9fad592874ef510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } } }
I20260809 03:34:28.326061 17242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:28.326267 17243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f3eec2bbf334054a9fad592874ef510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3eec2bbf334054a9fad592874ef510" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36113 } } }
I20260809 03:34:28.326370 17243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:28.326570 17256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:28.329660 17256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:28.334954 17256 catalog_manager.cc:1347] Generated new cluster ID: b33ff0fa19134766b5dff1482f3befc9
I20260809 03:34:28.335009 17256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:28.346284 17256 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:28.347110 17256 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:28.353818 17256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3eec2bbf334054a9fad592874ef510: Generated new TSK 0
I20260809 03:34:28.354413 17256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:28.361531 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17123
W20260809 03:34:28.371513 17100 connection.cc:535] client connection to 127.0.0.1:36113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36113): Connection reset by peer (error 104)
2026-08-09T03:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:28 dist-test-slave-cvwc krb5kdc[17275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:28 dist-test-slave-cvwc krb5kdc[17275](info): set up 2 sockets
Aug 09 03:34:28 dist-test-slave-cvwc krb5kdc[17275](info): commencing operation
krb5kdc: starting...
W20260809 03:34:29.490337 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:29 dist-test-slave-cvwc krb5kdc[17275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 03:34:29.604336 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:29.713198 17295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:29.713442 17295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:29.713497 17295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:29.717370 17295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:29.717427 17295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:29.717453 17295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:29.717473 17295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:29.717500 17295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:29.722643 17295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:29.723625 17295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:29.724640 17295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:29.732573 17302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:29.732626 17295 server_base.cc:1034] running on GCE node
W20260809 03:34:29.732551 17303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:29.732542 17306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:29.733314 17295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:29.734412 17295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:29.735574 17295 hybrid_clock.cc:648] HybridClock initialized: now 1786246469735547 us; error 35 us; skew 500 ppm
Aug 09 03:34:29 dist-test-slave-cvwc krb5kdc[17275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:29.742910 17295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 03:34:29.743785 17295 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260809 03:34:29.744313 17295 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:29.744375 17295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:29.744576 17295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:29.746618 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "d5ddcf755a6a47c7b82286e4a60349f8"
format_stamp: "Formatted at 2026-08-09 03:34:29 on dist-test-slave-cvwc"
server_key: "839f438a10544853e472c6a03cf636b2"
server_key_iv: "887825a9da69ca1153e6bf48661068ac"
server_key_version: "encryptionkey@0"
I20260809 03:34:29.747148 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "d5ddcf755a6a47c7b82286e4a60349f8"
format_stamp: "Formatted at 2026-08-09 03:34:29 on dist-test-slave-cvwc"
server_key: "839f438a10544853e472c6a03cf636b2"
server_key_iv: "887825a9da69ca1153e6bf48661068ac"
server_key_version: "encryptionkey@0"
I20260809 03:34:29.750717 17295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 03:34:29.752967 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:29.754204 17295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:29.754319 17295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "d5ddcf755a6a47c7b82286e4a60349f8"
format_stamp: "Formatted at 2026-08-09 03:34:29 on dist-test-slave-cvwc"
server_key: "839f438a10544853e472c6a03cf636b2"
server_key_iv: "887825a9da69ca1153e6bf48661068ac"
server_key_version: "encryptionkey@0"
I20260809 03:34:29.754413 17295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:29.758778 17295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:29.761478 17295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:29.761648 17295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:29.768154 17295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34531
I20260809 03:34:29.768160 17392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34531 every 8 connection(s)
I20260809 03:34:29.769029 17295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:29.769548 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17295
I20260809 03:34:29.769686 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:29.769994 16249 external_mini_cluster.cc:1428] Setting key a9b569a03a7e6279ce58ec8a16dc1c98
I20260809 03:34:29.771842 17395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:29.774864 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:29.771313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:37435 (local address 127.0.0.1:34531)
0809 03:34:29.771702 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:29.771718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:29.771752 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:34:29.772380 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:29.772406 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:29.772673 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:29.772944 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:29.773736 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:29.773743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:29.773758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:29.773769 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:29.773821 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:29.773840 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:29.773949 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:29.774051 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 03:34:29.774187 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260809 03:34:29.777562 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:29.778049 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8: Bootstrap starting.
I20260809 03:34:29.780644 17395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:29.781265 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:29.779848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:37649 (local address 127.0.0.1:34531)
0809 03:34:29.780004 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:29.780008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:29.780082 (+ 74us) server_negotiation.cc:407] Connection header received
0809 03:34:29.780191 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:29.780193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:29.780218 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:29.780294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:29.780916 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:29.780919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:29.780921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:29.780931 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:29.780956 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:29.780966 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:29.781032 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:29.781117 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 03:34:29.781147 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 03:34:29.781527 17395 log.cc:824] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:29.783267 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8: No bootstrap required, opened a new log
I20260809 03:34:29.786664 17395 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260809 03:34:29.786883 17395 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:29.786942 17395 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ddcf755a6a47c7b82286e4a60349f8, State: Initialized, Role: FOLLOWER
I20260809 03:34:29.787333 17395 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260809 03:34:29.787458 17395 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:29.787523 17395 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:29.787617 17395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:29.788419 17395 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260809 03:34:29.788726 17395 leader_election.cc:302] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ddcf755a6a47c7b82286e4a60349f8; no voters:
I20260809 03:34:29.788998 17395 leader_election.cc:288] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:29.789124 17408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:29.789379 17408 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [term 1 LEADER]: Becoming Leader. State: Replica: d5ddcf755a6a47c7b82286e4a60349f8, State: Running, Role: LEADER
I20260809 03:34:29.789666 17408 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260809 03:34:29.790176 17395 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:29.791342 17409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260809 03:34:29.791467 17409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:29.791854 17419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:29.791816 17410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ddcf755a6a47c7b82286e4a60349f8. Latest consensus state: current_term: 1 leader_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ddcf755a6a47c7b82286e4a60349f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260809 03:34:29.792030 17410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:29.795034 17419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:29.800467 17419 catalog_manager.cc:1347] Generated new cluster ID: f0376671386b42c98d0c28bc2938e432
I20260809 03:34:29.800531 17419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:29.826123 17419 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:29.826426 17419 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:29.850859 17419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ddcf755a6a47c7b82286e4a60349f8: Generated new TSK 0
I20260809 03:34:29.851708 17419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:29.859140 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17295
2026-08-09T03:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:29 dist-test-slave-cvwc krb5kdc[17437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:29 dist-test-slave-cvwc krb5kdc[17437](info): set up 2 sockets
Aug 09 03:34:29 dist-test-slave-cvwc krb5kdc[17437](info): commencing operation
krb5kdc: starting...
W20260809 03:34:31.001636 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:31.118009 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41645
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:31.225282 17458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:31.225524 17458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:31.225562 17458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:31.229382 17458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:31.229430 17458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:31.229449 17458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:31.229463 17458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:31.229488 17458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:31.234453 17458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:31.235343 17458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:31.236260 17458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:31.244087 17469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:31.244110 17473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:31.244096 17471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:31.244251 17458 server_base.cc:1034] running on GCE node
I20260809 03:34:31.244867 17458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:31.245875 17458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:31.247035 17458 hybrid_clock.cc:648] HybridClock initialized: now 1786246471247022 us; error 24 us; skew 500 ppm
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246471, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:31.249518 17458 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:31.250355 17458 webserver.cc:460] Webserver started at http://127.15.222.126:40589/ using document root <none> and password file <none>
I20260809 03:34:31.250852 17458 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:31.250921 17458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:31.251108 17458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:31.253067 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "8c63e32dfc164d128dca22b73966affc"
format_stamp: "Formatted at 2026-08-09 03:34:31 on dist-test-slave-cvwc"
server_key: "54b7e6c3139abe2ba677cdc21c7d02c8"
server_key_iv: "4eec414612b30b384430e41699dae529"
server_key_version: "encryptionkey@0"
I20260809 03:34:31.253633 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "8c63e32dfc164d128dca22b73966affc"
format_stamp: "Formatted at 2026-08-09 03:34:31 on dist-test-slave-cvwc"
server_key: "54b7e6c3139abe2ba677cdc21c7d02c8"
server_key_iv: "4eec414612b30b384430e41699dae529"
server_key_version: "encryptionkey@0"
I20260809 03:34:31.257082 17458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 03:34:31.259322 17484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:31.260448 17458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 03:34:31.260560 17458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "8c63e32dfc164d128dca22b73966affc"
format_stamp: "Formatted at 2026-08-09 03:34:31 on dist-test-slave-cvwc"
server_key: "54b7e6c3139abe2ba677cdc21c7d02c8"
server_key_iv: "4eec414612b30b384430e41699dae529"
server_key_version: "encryptionkey@0"
I20260809 03:34:31.260644 17458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:31.281427 17458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:31.284195 17458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:31.284370 17458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:31.291046 17458 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41645
I20260809 03:34:31.291065 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41645 every 8 connection(s)
I20260809 03:34:31.291937 17458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:31.293453 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17458
I20260809 03:34:31.293593 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:31.293895 16249 external_mini_cluster.cc:1428] Setting key 7e9dcce939b094018c5de7e8365728e2
I20260809 03:34:31.294860 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:31.301291 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc: Bootstrap starting.
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:31.304034 17574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:31.304867 17574 log.cc:824] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:31.306974 17574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc: No bootstrap required, opened a new log
I20260809 03:34:31.308007 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:31.295186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44679 (local address 127.15.222.126:41645)
0809 03:34:31.295581 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:31.295596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:31.295630 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:34:31.296240 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:31.296257 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:31.296573 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:31.296830 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:31.297769 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:31.298770 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:31.300013 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:31.300452 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:31.300564 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:31.303521 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:31.303526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:31.303538 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:31.303564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:31.305200 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:31.305755 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:31.305762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:31.305768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:31.305876 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:31.306228 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:31.306230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:31.306231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:31.306584 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:31.306730 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:31.307110 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 03:34:31.307319 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260809 03:34:31.310190 17574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } }
I20260809 03:34:31.310477 17574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:31.310549 17574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c63e32dfc164d128dca22b73966affc, State: Initialized, Role: FOLLOWER
I20260809 03:34:31.310982 17574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } }
I20260809 03:34:31.311137 17574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:31.311218 17574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:31.311264 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:31.311337 17574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:31.312168 17574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } }
I20260809 03:34:31.312538 17574 leader_election.cc:302] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c63e32dfc164d128dca22b73966affc; no voters:
I20260809 03:34:31.312837 17574 leader_election.cc:288] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:31.312922 17591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:31.313140 17591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [term 1 LEADER]: Becoming Leader. State: Replica: 8c63e32dfc164d128dca22b73966affc, State: Running, Role: LEADER
I20260809 03:34:31.313442 17591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } }
I20260809 03:34:31.314105 17574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:31.314627 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c63e32dfc164d128dca22b73966affc. Latest consensus state: current_term: 1 leader_uuid: "8c63e32dfc164d128dca22b73966affc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } } }
I20260809 03:34:31.314764 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:31.315809 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c63e32dfc164d128dca22b73966affc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c63e32dfc164d128dca22b73966affc" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41645 } } }
I20260809 03:34:31.315973 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:31.316761 17600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:31.320622 17600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:31.321174 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:31.313102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39751 (local address 127.15.222.126:41645)
0809 03:34:31.313300 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:31.313304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:31.313637 (+ 333us) server_negotiation.cc:407] Connection header received
0809 03:34:31.313794 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:31.313797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:31.313835 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:31.313907 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:31.314974 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:31.315692 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:31.316734 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:31.317013 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:31.317101 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:31.318380 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:31.318383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:31.318386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:31.318410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:31.319626 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:31.320040 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:31.320043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:31.320045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:31.320116 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:31.320411 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:31.320414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:31.320416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:31.320569 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:31.320687 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:31.320954 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 03:34:31.321034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 03:34:31.325948 17600 catalog_manager.cc:1347] Generated new cluster ID: 4c7f681b4eb548febd74e8df645871a1
I20260809 03:34:31.326012 17600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:31.335988 17600 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:31.336762 17600 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:31.349890 17600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c63e32dfc164d128dca22b73966affc: Generated new TSK 0
I20260809 03:34:31.350440 17600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:31.355712 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17458
W20260809 03:34:31.372619 17435 connection.cc:535] client connection to 127.15.222.126:41645 recv error: Network error: failed to read from TLS socket (remote: 127.15.222.126:41645): Connection reset by peer (error 104)
2026-08-09T03:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17631](info): set up 2 sockets
Aug 09 03:34:31 dist-test-slave-cvwc krb5kdc[17631](info): commencing operation
krb5kdc: starting...
W20260809 03:34:32.504141 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:32 dist-test-slave-cvwc krb5kdc[17631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:32.618408 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37559
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:32.726668 17650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:32.726918 17650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:32.726958 17650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:32.730818 17650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:32.730863 17650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:32.730881 17650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:32.730901 17650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:32.730916 17650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:32.735877 17650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:37559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:32.736749 17650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:32.737643 17650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:32.745662 17658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:32.745644 17657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:32.745774 17650 server_base.cc:1034] running on GCE node
W20260809 03:34:32.745645 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:32.746496 17650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:32.747488 17650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:32.748646 17650 hybrid_clock.cc:648] HybridClock initialized: now 1786246472748628 us; error 29 us; skew 500 ppm
Aug 09 03:34:32 dist-test-slave-cvwc krb5kdc[17631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246472, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:32.751267 17650 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:32.752075 17650 webserver.cc:460] Webserver started at http://127.15.222.126:44387/ using document root <none> and password file <none>
I20260809 03:34:32.752591 17650 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:32.752656 17650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:32.752836 17650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:32.754820 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "89b16de0a8d34090854dc5d4692f78ce"
format_stamp: "Formatted at 2026-08-09 03:34:32 on dist-test-slave-cvwc"
server_key: "b4d153323815c4a4ccc3866bd654d255"
server_key_iv: "9d117cd4aa99e239e96da8a20d459365"
server_key_version: "encryptionkey@0"
I20260809 03:34:32.755340 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "89b16de0a8d34090854dc5d4692f78ce"
format_stamp: "Formatted at 2026-08-09 03:34:32 on dist-test-slave-cvwc"
server_key: "b4d153323815c4a4ccc3866bd654d255"
server_key_iv: "9d117cd4aa99e239e96da8a20d459365"
server_key_version: "encryptionkey@0"
I20260809 03:34:32.759023 17650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:32.761243 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:32.762445 17650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:32.762558 17650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "89b16de0a8d34090854dc5d4692f78ce"
format_stamp: "Formatted at 2026-08-09 03:34:32 on dist-test-slave-cvwc"
server_key: "b4d153323815c4a4ccc3866bd654d255"
server_key_iv: "9d117cd4aa99e239e96da8a20d459365"
server_key_version: "encryptionkey@0"
I20260809 03:34:32.762653 17650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:32.783711 17650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:32.786753 17650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:32.786931 17650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:32.793351 17650 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:37559
I20260809 03:34:32.793399 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:37559 every 8 connection(s)
I20260809 03:34:32.794262 17650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:32.796882 17762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:32.802207 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: Bootstrap starting.
I20260809 03:34:32.803763 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17650
I20260809 03:34:32.803877 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:32.804136 16249 external_mini_cluster.cc:1428] Setting key 9efb7918123fee8ee6e9ac41fc7ef87f
I20260809 03:34:32.804564 17762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:32.805403 17762 log.cc:824] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:32.807412 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: No bootstrap required, opened a new log
I20260809 03:34:32.811270 17762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } }
I20260809 03:34:32.811489 17762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:32.811551 17762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b16de0a8d34090854dc5d4692f78ce, State: Initialized, Role: FOLLOWER
I20260809 03:34:32.812044 17762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } }
I20260809 03:34:32.812189 17762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:32.812255 17762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:32.812352 17762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:32.812667 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:32.805141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42071 (local address 127.15.222.126:37559)
0809 03:34:32.805530 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:32.805550 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:32.805589 (+ 39us) server_negotiation.cc:407] Connection header received
0809 03:34:32.806363 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:32.806383 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:32.806757 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:32.807044 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:32.807360 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:32.808326 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:32.809522 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:32.810003 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:32.810113 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:32.810955 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:32.810960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:32.810973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:32.811326 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:32.811377 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:32.811414 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:32.811527 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:32.811766 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 03:34:32.811877 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260809 03:34:32.813192 17762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } }
I20260809 03:34:32.813515 17762 leader_election.cc:302] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b16de0a8d34090854dc5d4692f78ce; no voters:
I20260809 03:34:32.813855 17762 leader_election.cc:288] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:32.813931 17771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:32.814139 17771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [term 1 LEADER]: Becoming Leader. State: Replica: 89b16de0a8d34090854dc5d4692f78ce, State: Running, Role: LEADER
I20260809 03:34:32.814433 17771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } }
I20260809 03:34:32.815027 17762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:32.815791 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:32.816138 17774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b16de0a8d34090854dc5d4692f78ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } } }
I20260809 03:34:32.816140 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b16de0a8d34090854dc5d4692f78ce. Latest consensus state: current_term: 1 leader_uuid: "89b16de0a8d34090854dc5d4692f78ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b16de0a8d34090854dc5d4692f78ce" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 37559 } } }
I20260809 03:34:32.816273 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:32.816273 17774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:32.819953 17798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:32.820019 17798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:32.820094 17788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:32.820817 17788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:32.822270 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:32.817922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42941 (local address 127.15.222.126:37559)
0809 03:34:32.818019 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:32.818023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:32.818315 (+ 292us) server_negotiation.cc:407] Connection header received
0809 03:34:32.818411 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:32.818414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:32.818455 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:32.818542 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:32.818803 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:32.819537 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:32.820596 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:32.820908 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:32.821007 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:32.821826 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:32.821830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:32.821832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:32.821845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:32.821871 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:32.821883 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:32.821970 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:32.822112 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 03:34:32.822144 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 03:34:32.826295 17788 catalog_manager.cc:1347] Generated new cluster ID: b2a0b355530e47d2ad45aabb455ca241
I20260809 03:34:32.826352 17788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:32.841473 17788 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:32.842372 17788 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:32.849649 17788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b16de0a8d34090854dc5d4692f78ce: Generated new TSK 0
I20260809 03:34:32.850448 17788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:32.860502 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17650
W20260809 03:34:32.874660 17627 connection.cc:535] client connection to 127.15.222.126:37559 recv error: Network error: failed to read from TLS socket (remote: 127.15.222.126:37559): Connection reset by peer (error 104)
2026-08-09T03:34:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:32 dist-test-slave-cvwc krb5kdc[17814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:32 dist-test-slave-cvwc krb5kdc[17814](info): set up 2 sockets
Aug 09 03:34:32 dist-test-slave-cvwc krb5kdc[17814](info): commencing operation
krb5kdc: starting...
W20260809 03:34:34.009946 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:34 dist-test-slave-cvwc krb5kdc[17814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:34.126091 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35159
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:34.232522 17837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:34.232754 17837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:34.232792 17837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:34.236627 17837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:34.236675 17837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:34.236693 17837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:34.236707 17837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:34.236726 17837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:34.241667 17837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:35159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:34.242564 17837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:34.243462 17837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:34.251394 17849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:34.251500 17837 server_base.cc:1034] running on GCE node
W20260809 03:34:34.251405 17846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:34.251410 17847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:34.252213 17837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:34.253177 17837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:34.254328 17837 hybrid_clock.cc:648] HybridClock initialized: now 1786246474254312 us; error 25 us; skew 500 ppm
Aug 09 03:34:34 dist-test-slave-cvwc krb5kdc[17814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246474, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:34.256835 17837 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:34.257647 17837 webserver.cc:460] Webserver started at http://127.15.222.126:38663/ using document root <none> and password file <none>
I20260809 03:34:34.258177 17837 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:34.258236 17837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:34.258425 17837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:34.260432 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "bc4577cd24af460e800c54bb9d959cd1"
format_stamp: "Formatted at 2026-08-09 03:34:34 on dist-test-slave-cvwc"
server_key: "2846fda956f780b859f54f1dacabfab7"
server_key_iv: "17d3843030e51cc237fd88f6c7308531"
server_key_version: "encryptionkey@0"
I20260809 03:34:34.260985 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "bc4577cd24af460e800c54bb9d959cd1"
format_stamp: "Formatted at 2026-08-09 03:34:34 on dist-test-slave-cvwc"
server_key: "2846fda956f780b859f54f1dacabfab7"
server_key_iv: "17d3843030e51cc237fd88f6c7308531"
server_key_version: "encryptionkey@0"
I20260809 03:34:34.264474 17837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 03:34:34.266716 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:34.267863 17837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 03:34:34.267992 17837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "bc4577cd24af460e800c54bb9d959cd1"
format_stamp: "Formatted at 2026-08-09 03:34:34 on dist-test-slave-cvwc"
server_key: "2846fda956f780b859f54f1dacabfab7"
server_key_iv: "17d3843030e51cc237fd88f6c7308531"
server_key_version: "encryptionkey@0"
I20260809 03:34:34.268082 17837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:34.272564 17837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:34.275245 17837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:34.275409 17837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:34.281857 17837 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:35159
I20260809 03:34:34.281885 17953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:35159 every 8 connection(s)
I20260809 03:34:34.282738 17837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:34.285347 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:34.290736 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1: Bootstrap starting.
I20260809 03:34:34.291175 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 17837
I20260809 03:34:34.291299 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:34.291567 16249 external_mini_cluster.cc:1428] Setting key 026cd7837cddaa9273df65378681d09d
I20260809 03:34:34.293267 17954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:34.293983 17954 log.cc:824] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:34.295837 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1: No bootstrap required, opened a new log
I20260809 03:34:34.296371 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:34.292756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52247 (local address 127.15.222.126:35159)
0809 03:34:34.293176 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:34.293190 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:34.293225 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:34:34.293889 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:34.293907 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:34.294191 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:34.294482 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:34.295218 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:34.295222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:34.295234 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:34.295244 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:34.295297 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:34.295316 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:34.295407 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:34.295546 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 03:34:34.295693 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":135,"threads_started":1}
I20260809 03:34:34.299058 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:34.299701 17954 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } }
I20260809 03:34:34.299904 17954 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:34.299968 17954 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4577cd24af460e800c54bb9d959cd1, State: Initialized, Role: FOLLOWER
I20260809 03:34:34.300452 17954 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } }
I20260809 03:34:34.300587 17954 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:34.300688 17954 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:34.300797 17954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:34.301824 17954 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } }
I20260809 03:34:34.302172 17954 leader_election.cc:302] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc4577cd24af460e800c54bb9d959cd1; no voters:
I20260809 03:34:34.302570 17954 leader_election.cc:288] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:34.302634 17976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:34.302760 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:34.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38193 (local address 127.15.222.126:35159)
0809 03:34:34.301362 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:34.301366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:34.301379 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:34.301412 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:34.301415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:34.301443 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:34.301523 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:34.302156 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:34.302158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:34.302161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:34.302173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:34.302200 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:34.302213 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:34.302300 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:34.302392 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 03:34:34.302430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 03:34:34.302839 17976 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [term 1 LEADER]: Becoming Leader. State: Replica: bc4577cd24af460e800c54bb9d959cd1, State: Running, Role: LEADER
I20260809 03:34:34.303128 17976 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } }
I20260809 03:34:34.303887 17954 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:34.304119 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc4577cd24af460e800c54bb9d959cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } } }
I20260809 03:34:34.304124 17976 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4577cd24af460e800c54bb9d959cd1. Latest consensus state: current_term: 1 leader_uuid: "bc4577cd24af460e800c54bb9d959cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc4577cd24af460e800c54bb9d959cd1" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 35159 } } }
I20260809 03:34:34.304262 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:34.304262 17976 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:34.304607 17980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:34.307183 17980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:34.311769 17980 catalog_manager.cc:1347] Generated new cluster ID: 027839653f364b9a94923f58651a75c0
I20260809 03:34:34.311828 17980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:34.334897 17980 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:34.335182 17980 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:34.339617 17980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc4577cd24af460e800c54bb9d959cd1: Generated new TSK 0
I20260809 03:34:34.340330 17980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:34.344393 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 17837
2026-08-09T03:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:34 dist-test-slave-cvwc krb5kdc[18010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:34 dist-test-slave-cvwc krb5kdc[18010](info): set up 2 sockets
Aug 09 03:34:34 dist-test-slave-cvwc krb5kdc[18010](info): commencing operation
krb5kdc: starting...
W20260809 03:34:35.485663 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:35.599756 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46791
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:35.707423 18043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:35.707674 18043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:35.707732 18043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:35.711725 18043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:35.711784 18043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:35.711809 18043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:35.711829 18043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:35.711851 18043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:35.717289 18043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:35.718327 18043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:35.719350 18043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:35.727475 18055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:35.727491 18051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:35.727475 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:35.727720 18043 server_base.cc:1034] running on GCE node
I20260809 03:34:35.728374 18043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:35.729374 18043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:35.730525 18043 hybrid_clock.cc:648] HybridClock initialized: now 1786246475730503 us; error 33 us; skew 500 ppm
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246475, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:35.735680 18043 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:35.736563 18043 webserver.cc:460] Webserver started at http://127.15.222.126:42773/ using document root <none> and password file <none>
I20260809 03:34:35.737084 18043 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:35.737149 18043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:35.737335 18043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:35.739406 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "e91a6acdcd2a4dbcb52bfb7081258b01"
format_stamp: "Formatted at 2026-08-09 03:34:35 on dist-test-slave-cvwc"
server_key: "e26ad87e7a4ebf67b0091d7d9cbddfcf"
server_key_iv: "f68722788250cf168c3bb5542c70fb53"
server_key_version: "encryptionkey@0"
I20260809 03:34:35.739938 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "e91a6acdcd2a4dbcb52bfb7081258b01"
format_stamp: "Formatted at 2026-08-09 03:34:35 on dist-test-slave-cvwc"
server_key: "e26ad87e7a4ebf67b0091d7d9cbddfcf"
server_key_iv: "f68722788250cf168c3bb5542c70fb53"
server_key_version: "encryptionkey@0"
I20260809 03:34:35.743464 18043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 03:34:35.745657 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:35.746824 18043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:35.746937 18043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "e91a6acdcd2a4dbcb52bfb7081258b01"
format_stamp: "Formatted at 2026-08-09 03:34:35 on dist-test-slave-cvwc"
server_key: "e26ad87e7a4ebf67b0091d7d9cbddfcf"
server_key_iv: "f68722788250cf168c3bb5542c70fb53"
server_key_version: "encryptionkey@0"
I20260809 03:34:35.747027 18043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:35.758850 18043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:35.761766 18043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:35.761932 18043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:35.768565 18043 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:46791
I20260809 03:34:35.768580 18147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:46791 every 8 connection(s)
I20260809 03:34:35.769474 18043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:35.772150 18148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:35.775218 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18043
I20260809 03:34:35.775364 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:35.775682 16249 external_mini_cluster.cc:1428] Setting key c840f2545064954d9a233757b697f5e5
I20260809 03:34:35.778049 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01: Bootstrap starting.
I20260809 03:34:35.780426 18148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:35.781244 18148 log.cc:824] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:35.783190 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01: No bootstrap required, opened a new log
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:35.786792 18148 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } }
I20260809 03:34:35.787017 18148 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:35.787076 18148 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91a6acdcd2a4dbcb52bfb7081258b01, State: Initialized, Role: FOLLOWER
I20260809 03:34:35.787563 18148 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } }
I20260809 03:34:35.787725 18148 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:35.787799 18148 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:35.787914 18148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:35.788862 18148 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } }
I20260809 03:34:35.789196 18148 leader_election.cc:302] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91a6acdcd2a4dbcb52bfb7081258b01; no voters:
I20260809 03:34:35.789439 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:35.777075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45837 (local address 127.15.222.126:46791)
0809 03:34:35.777452 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:35.777467 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:35.777502 (+ 35us) server_negotiation.cc:407] Connection header received
0809 03:34:35.778167 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:35.778187 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:35.778545 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:35.778809 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:35.779675 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:35.780680 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:35.781785 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:35.782254 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:35.782405 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:35.784783 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:35.784788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:35.784801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:35.784828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:35.786410 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:35.786824 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:35.786829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:35.786834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:35.786947 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:35.787222 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:35.787226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:35.787228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:35.787739 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:35.787944 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:35.788350 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 03:34:35.788630 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260809 03:34:35.789469 18148 leader_election.cc:288] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:35.789564 18157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:35.789808 18157 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [term 1 LEADER]: Becoming Leader. State: Replica: e91a6acdcd2a4dbcb52bfb7081258b01, State: Running, Role: LEADER
I20260809 03:34:35.790082 18157 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } }
I20260809 03:34:35.791273 18159 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } } }
I20260809 03:34:35.791405 18159 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:35.791569 18148 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:35.792085 18158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91a6acdcd2a4dbcb52bfb7081258b01. Latest consensus state: current_term: 1 leader_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91a6acdcd2a4dbcb52bfb7081258b01" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 46791 } } }
I20260809 03:34:35.792479 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:35.792699 18158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:35.792781 18164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:35.795868 18164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:35.801183 18164 catalog_manager.cc:1347] Generated new cluster ID: 0fa3dcd6c2e1425380881722ebc46841
I20260809 03:34:35.801287 18164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:35.806309 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:35.794599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53469 (local address 127.15.222.126:46791)
0809 03:34:35.794704 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:35.794708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:35.798009 (+ 3301us) server_negotiation.cc:407] Connection header received
0809 03:34:35.798112 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:35.798115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:35.798155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:35.798215 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:35.798818 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:35.800118 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:35.801275 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:35.801645 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:35.801776 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:35.803210 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:35.803214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:35.803217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:35.803242 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:35.804785 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:35.805195 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:35.805198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:35.805200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:35.805286 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:35.805511 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:35.805514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:35.805516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:35.805733 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:35.805857 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:35.806056 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 03:34:35.806162 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260809 03:34:35.814034 18164 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:35.814847 18164 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:35.824513 18164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91a6acdcd2a4dbcb52bfb7081258b01: Generated new TSK 0
I20260809 03:34:35.825215 18164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:35.830658 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18043
2026-08-09T03:34:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18190](info): set up 2 sockets
Aug 09 03:34:35 dist-test-slave-cvwc krb5kdc[18190](info): commencing operation
krb5kdc: starting...
W20260809 03:34:37.005820 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:37 dist-test-slave-cvwc krb5kdc[18190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:37.122701 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41233
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:37.230986 18212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:37.231215 18212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:37.231266 18212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:37.235194 18212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:37.235245 18212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:37.235270 18212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:37.235301 18212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:37.235327 18212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:37.240367 18212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:37.241278 18212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:37.242218 18212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:37.249958 18220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:37.249958 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:37.250017 18212 server_base.cc:1034] running on GCE node
W20260809 03:34:37.249980 18222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:37.250725 18212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:37.251735 18212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:37.252905 18212 hybrid_clock.cc:648] HybridClock initialized: now 1786246477252889 us; error 25 us; skew 500 ppm
Aug 09 03:34:37 dist-test-slave-cvwc krb5kdc[18190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246477, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:37.255578 18212 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:37.256417 18212 webserver.cc:460] Webserver started at http://127.15.222.126:39887/ using document root <none> and password file <none>
I20260809 03:34:37.256933 18212 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:37.256994 18212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:37.257181 18212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:37.259171 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "da461d7030d5422caeefeddf7b64952e"
format_stamp: "Formatted at 2026-08-09 03:34:37 on dist-test-slave-cvwc"
server_key: "29942d2445b998e01b75794760ba0080"
server_key_iv: "b2d9638abd21043f2d9e191da32d126b"
server_key_version: "encryptionkey@0"
I20260809 03:34:37.259697 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "da461d7030d5422caeefeddf7b64952e"
format_stamp: "Formatted at 2026-08-09 03:34:37 on dist-test-slave-cvwc"
server_key: "29942d2445b998e01b75794760ba0080"
server_key_iv: "b2d9638abd21043f2d9e191da32d126b"
server_key_version: "encryptionkey@0"
I20260809 03:34:37.263276 18212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:37.265569 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:37.266716 18212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:37.266851 18212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "da461d7030d5422caeefeddf7b64952e"
format_stamp: "Formatted at 2026-08-09 03:34:37 on dist-test-slave-cvwc"
server_key: "29942d2445b998e01b75794760ba0080"
server_key_iv: "b2d9638abd21043f2d9e191da32d126b"
server_key_version: "encryptionkey@0"
I20260809 03:34:37.266940 18212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:37.276160 18212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:37.278707 18212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:37.278877 18212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:37.285193 18212 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41233
I20260809 03:34:37.285238 18315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41233 every 8 connection(s)
I20260809 03:34:37.286124 18212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:37.288085 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18212
I20260809 03:34:37.288228 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:37.288558 16249 external_mini_cluster.cc:1428] Setting key 03be070e6f93b2ca315f536d4a902aaa
I20260809 03:34:37.289027 18316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:37.295395 18316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: Bootstrap starting.
I20260809 03:34:37.297113 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:37.289832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49449 (local address 127.15.222.126:41233)
0809 03:34:37.290209 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:37.290224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:37.290258 (+ 34us) server_negotiation.cc:407] Connection header received
0809 03:34:37.290895 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:37.290912 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:37.291228 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:37.291485 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:37.291841 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:37.292819 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:37.293981 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:37.294466 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:37.294605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:37.295431 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:37.295436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:37.295450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:37.295870 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:37.295929 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:37.295956 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:37.296037 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:37.296183 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 03:34:37.296386 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260809 03:34:37.297600 18316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:37.298367 18316 log.cc:824] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:37.299892 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:37.300421 18316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: No bootstrap required, opened a new log
I20260809 03:34:37.304129 18316 raft_consensus.cc:348] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } }
I20260809 03:34:37.304335 18316 raft_consensus.cc:374] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:37.304370 18316 raft_consensus.cc:729] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da461d7030d5422caeefeddf7b64952e, State: Initialized, Role: FOLLOWER
I20260809 03:34:37.304780 18316 consensus_queue.cc:260] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } }
I20260809 03:34:37.304939 18316 raft_consensus.cc:388] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:37.305002 18316 raft_consensus.cc:482] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:37.305102 18316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:37.306005 18316 raft_consensus.cc:504] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } }
I20260809 03:34:37.306176 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:37.302013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56897 (local address 127.15.222.126:41233)
0809 03:34:37.302123 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:37.302127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:37.302318 (+ 191us) server_negotiation.cc:407] Connection header received
0809 03:34:37.302404 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:37.302407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:37.302447 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:37.302518 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:37.302714 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:37.303878 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:37.304693 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:37.305019 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:37.305124 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:37.305825 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:37.305828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:37.305831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:37.305844 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:37.305869 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:37.305882 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:37.305956 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:37.306016 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 03:34:37.306063 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:34:37.306356 18316 leader_election.cc:302] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da461d7030d5422caeefeddf7b64952e; no voters:
I20260809 03:34:37.306679 18316 leader_election.cc:288] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:37.306804 18332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:37.307258 18332 raft_consensus.cc:686] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [term 1 LEADER]: Becoming Leader. State: Replica: da461d7030d5422caeefeddf7b64952e, State: Running, Role: LEADER
I20260809 03:34:37.307542 18332 consensus_queue.cc:237] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } }
I20260809 03:34:37.307804 18316 sys_catalog.cc:564] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:37.309163 18333 sys_catalog.cc:455] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da461d7030d5422caeefeddf7b64952e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } } }
I20260809 03:34:37.309203 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [sys.catalog]: SysCatalogTable state changed. Reason: New leader da461d7030d5422caeefeddf7b64952e. Latest consensus state: current_term: 1 leader_uuid: "da461d7030d5422caeefeddf7b64952e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da461d7030d5422caeefeddf7b64952e" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41233 } } }
I20260809 03:34:37.309293 18333 sys_catalog.cc:458] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:37.309309 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:37.312184 18351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:37.312265 18351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:37.312363 18350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:37.313048 18350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:37.318307 18350 catalog_manager.cc:1347] Generated new cluster ID: 363c14eff026414d97c1a71c9b1e3eb9
I20260809 03:34:37.318367 18350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:37.329422 18350 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:37.330416 18350 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:37.337280 18350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da461d7030d5422caeefeddf7b64952e: Generated new TSK 0
I20260809 03:34:37.337885 18350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:37.341549 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18212
2026-08-09T03:34:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:37 dist-test-slave-cvwc krb5kdc[18367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:37 dist-test-slave-cvwc krb5kdc[18367](info): set up 2 sockets
Aug 09 03:34:37 dist-test-slave-cvwc krb5kdc[18367](info): commencing operation
krb5kdc: starting...
W20260809 03:34:38.485613 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:38 dist-test-slave-cvwc krb5kdc[18367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:38.597415 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:40581
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:38.704507 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:38.704753 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:38.704809 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:38.708638 18388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:38.708688 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:38.708714 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:38.708736 18388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:38.708765 18388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:38.713821 18388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:40581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:38.714756 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:38.715713 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:38.723567 18399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:38.723570 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:38.723570 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:38.723922 18388 server_base.cc:1034] running on GCE node
I20260809 03:34:38.724434 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:38.725373 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:38.726516 18388 hybrid_clock.cc:648] HybridClock initialized: now 1786246478726496 us; error 31 us; skew 500 ppm
Aug 09 03:34:38 dist-test-slave-cvwc krb5kdc[18367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246478, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:38.728995 18388 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:38.729804 18388 webserver.cc:460] Webserver started at http://127.15.222.126:45171/ using document root <none> and password file <none>
I20260809 03:34:38.730307 18388 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:38.730366 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:38.730556 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:38.732460 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "64bd6903099c43f7abd646a8fa189225"
format_stamp: "Formatted at 2026-08-09 03:34:38 on dist-test-slave-cvwc"
server_key: "a0ca9c93086dc565de18ae827986113c"
server_key_iv: "0ce99e255f3ab6baaf9a3e9f8ca7f0ec"
server_key_version: "encryptionkey@0"
I20260809 03:34:38.732972 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "64bd6903099c43f7abd646a8fa189225"
format_stamp: "Formatted at 2026-08-09 03:34:38 on dist-test-slave-cvwc"
server_key: "a0ca9c93086dc565de18ae827986113c"
server_key_iv: "0ce99e255f3ab6baaf9a3e9f8ca7f0ec"
server_key_version: "encryptionkey@0"
I20260809 03:34:38.736423 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 03:34:38.738626 18409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:38.739725 18388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:38.739837 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "64bd6903099c43f7abd646a8fa189225"
format_stamp: "Formatted at 2026-08-09 03:34:38 on dist-test-slave-cvwc"
server_key: "a0ca9c93086dc565de18ae827986113c"
server_key_iv: "0ce99e255f3ab6baaf9a3e9f8ca7f0ec"
server_key_version: "encryptionkey@0"
I20260809 03:34:38.739923 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:38.744227 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:38.746994 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:38.747155 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:38.753412 18388 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:40581
I20260809 03:34:38.753423 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:40581 every 8 connection(s)
I20260809 03:34:38.754245 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:38.756780 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:38.761968 18509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: Bootstrap starting.
I20260809 03:34:38.762221 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18388
I20260809 03:34:38.762334 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:38.762580 16249 external_mini_cluster.cc:1428] Setting key 8ae0b6b92247ef4ff43284a853ac3b16
I20260809 03:34:38.764303 18509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:38.765023 18509 log.cc:824] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:38.767000 18509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: No bootstrap required, opened a new log
I20260809 03:34:38.767333 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:38.763737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44039 (local address 127.15.222.126:40581)
0809 03:34:38.764132 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:38.764147 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:38.764183 (+ 36us) server_negotiation.cc:407] Connection header received
0809 03:34:38.764812 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:38.764831 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:38.765100 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:38.765414 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:38.766146 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:38.766150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:38.766163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:38.766173 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:38.766223 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:38.766242 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:38.766343 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:38.766463 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 03:34:38.766613 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260809 03:34:38.770133 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:38.770891 18509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } }
I20260809 03:34:38.771102 18509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 03:34:38.771139 18509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64bd6903099c43f7abd646a8fa189225, State: Initialized, Role: FOLLOWER
I20260809 03:34:38.771586 18509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } }
I20260809 03:34:38.771708 18509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 03:34:38.771790 18509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 03:34:38.771888 18509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:38.772744 18509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } }
I20260809 03:34:38.773101 18509 leader_election.cc:302] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64bd6903099c43f7abd646a8fa189225; no voters:
I20260809 03:34:38.773389 18509 leader_election.cc:288] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 03:34:38.773473 18525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:38.773598 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:38.772022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46999 (local address 127.15.222.126:40581)
0809 03:34:38.772136 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:38.772140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:38.772353 (+ 213us) server_negotiation.cc:407] Connection header received
0809 03:34:38.772479 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:38.772482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:38.772512 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:38.772574 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:38.773203 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:38.773207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:38.773209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 03:34:38.773220 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:38.773249 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 03:34:38.773261 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:38.773333 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:38.773406 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 03:34:38.773441 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 03:34:38.773695 18525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [term 1 LEADER]: Becoming Leader. State: Replica: 64bd6903099c43f7abd646a8fa189225, State: Running, Role: LEADER
I20260809 03:34:38.774493 18525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } }
I20260809 03:34:38.774677 18509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:38.776051 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64bd6903099c43f7abd646a8fa189225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } } }
I20260809 03:34:38.776181 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:38.776513 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64bd6903099c43f7abd646a8fa189225. Latest consensus state: current_term: 1 leader_uuid: "64bd6903099c43f7abd646a8fa189225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64bd6903099c43f7abd646a8fa189225" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 40581 } } }
I20260809 03:34:38.776592 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225 [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:38.778998 18553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:38.779074 18553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:38.779148 18544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:38.779843 18544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:38.784813 18544 catalog_manager.cc:1347] Generated new cluster ID: 4cf05460286347288917124bb1b87b4c
I20260809 03:34:38.784921 18544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:38.796403 18544 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:38.796685 18544 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:38.803519 18544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64bd6903099c43f7abd646a8fa189225: Generated new TSK 0
I20260809 03:34:38.804095 18544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 03:34:38.809608 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18388
2026-08-09T03:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1464 ms)
[----------] 18 tests from AuthTokenIssuingTest (26964 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:38 dist-test-slave-cvwc krb5kdc[18560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:38 dist-test-slave-cvwc krb5kdc[18560](info): set up 2 sockets
Aug 09 03:34:38 dist-test-slave-cvwc krb5kdc[18560](info): commencing operation
krb5kdc: starting...
W20260809 03:34:39.941632 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:40.058290 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41291
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:40.167212 18580 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:40.167475 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:40.167496 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:40.167544 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:40.171432 18580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:40.171481 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:40.171499 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:40.171528 18580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:40.171555 18580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:40.176554 18580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:40.177474 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:40.178442 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:40.186353 18591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.186388 18594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.186404 18590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:40.186529 18580 server_base.cc:1034] running on GCE node
I20260809 03:34:40.187167 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:40.188163 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:40.189308 18580 hybrid_clock.cc:648] HybridClock initialized: now 1786246480189286 us; error 34 us; skew 500 ppm
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:40.191874 18580 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:40.192710 18580 webserver.cc:460] Webserver started at http://127.15.222.126:41799/ using document root <none> and password file <none>
I20260809 03:34:40.193220 18580 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:40.193285 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:40.193491 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:40.195462 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "553330ebe2274be3a54942691812bac5"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0b854f20c0d458a95d83a0349fcf09ab"
server_key_iv: "0adc79066d588fbeb3b48ada771683df"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.195984 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "553330ebe2274be3a54942691812bac5"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0b854f20c0d458a95d83a0349fcf09ab"
server_key_iv: "0adc79066d588fbeb3b48ada771683df"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.199538 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:40.201689 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:40.202803 18580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:40.202913 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "553330ebe2274be3a54942691812bac5"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0b854f20c0d458a95d83a0349fcf09ab"
server_key_iv: "0adc79066d588fbeb3b48ada771683df"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.203001 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:40.212284 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:40.214754 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:40.214910 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:40.221218 18580 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:41291
I20260809 03:34:40.221458 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:41291 every 8 connection(s)
I20260809 03:34:40.222121 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:40.224118 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18580
I20260809 03:34:40.224237 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:40.224522 16249 external_mini_cluster.cc:1428] Setting key 21af650aeafe728377a98a1eb5e52381
I20260809 03:34:40.224902 18689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:40.228739 18689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.125@KRBTEST.COM" created.
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:40.243530 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.230928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.15.222.126:41291)
0809 03:34:40.231433 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.231461 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.231614 (+ 153us) server_negotiation.cc:407] Connection header received
0809 03:34:40.232845 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.232849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.233033 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.233178 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.233946 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.234864 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.235989 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.236562 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.236716 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.239170 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.239175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.239184 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.239204 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.240639 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.241060 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.241064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.241065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.241163 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.241435 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.241438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.241440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.241818 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.241934 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.242277 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.242516 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":392,"spinlock_wait_cycles":81920,"thread_start_us":143,"threads_started":1}
I20260809 03:34:40.243530 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.230928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41291 (local address 127.0.0.1:33630)
0809 03:34:40.231456 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.231508 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.232562 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.233227 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.233232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.233529 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.233815 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.233830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.234966 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.234969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.235863 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.235873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.236699 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.236702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.236834 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.237400 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.237421 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.239041 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.240735 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.240744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.240754 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.240966 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.241244 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.241247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.241249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.241356 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.241918 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.241924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.242060 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.242203 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.242574 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":106,"spinlock_wait_cycles":2944,"thread_start_us":129,"threads_started":1}
I20260809 03:34:40.245899 18689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.246542 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.246298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38568)
0809 03:34:40.246401 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.246483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 03:34:40.246701 18609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
I20260809 03:34:40.247511 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.247309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38576)
0809 03:34:40.247406 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.247459 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:34:40.247639 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45865: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
I20260809 03:34:40.249450 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:45865
--webserver_interface=127.15.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:34:40.296885 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } attempt: 1
I20260809 03:34:40.297798 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.297492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38582)
0809 03:34:40.297635 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.297695 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 03:34:40.298492 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.298267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38586)
0809 03:34:40.298363 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.298412 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:34:40.298604 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45865: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
W20260809 03:34:40.357873 18701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:40.358146 18701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:40.358179 18701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:40.358239 18701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:40.362174 18701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:40.362228 18701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:40.362253 18701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:40.362273 18701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:40.362301 18701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:40.366758 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } attempt: 2
I20260809 03:34:40.367502 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.367220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38588)
0809 03:34:40.367358 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.367417 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 03:34:40.367808 18701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:45865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:40.368191 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.367955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38604)
0809 03:34:40.368097 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.368134 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260809 03:34:40.368309 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:45865: Network error: Client connection negotiation failed: client connection to 127.15.222.125:45865: connect: Connection refused (error 111)
I20260809 03:34:40.368868 18701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:40.369863 18701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:40.378111 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.378116 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.378147 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:40.378299 18701 server_base.cc:1034] running on GCE node
I20260809 03:34:40.378965 18701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:40.380028 18701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:40.381187 18701 hybrid_clock.cc:648] HybridClock initialized: now 1786246480381164 us; error 33 us; skew 500 ppm
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:40.383817 18701 init.cc:377] Logged in from keytab as kudu/127.15.222.125@KRBTEST.COM (short username kudu)
I20260809 03:34:40.384696 18701 webserver.cc:460] Webserver started at http://127.15.222.125:41917/ using document root <none> and password file <none>
I20260809 03:34:40.385223 18701 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:40.385288 18701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:40.385483 18701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:40.387509 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data/instance:
uuid: "212c057ec49e4d2586a911f1b7ede14a"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0f1e2934afa5a3eb258f815a562f49d5"
server_key_iv: "7e06642afdcd665e5de17d76d5932b9d"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.388056 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal/instance:
uuid: "212c057ec49e4d2586a911f1b7ede14a"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0f1e2934afa5a3eb258f815a562f49d5"
server_key_iv: "7e06642afdcd665e5de17d76d5932b9d"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.391513 18701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 03:34:40.393688 18724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:40.394822 18701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:40.394948 18701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "212c057ec49e4d2586a911f1b7ede14a"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "0f1e2934afa5a3eb258f815a562f49d5"
server_key_iv: "7e06642afdcd665e5de17d76d5932b9d"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.395037 18701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:40.407078 18701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:40.409528 18701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:40.409680 18701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:40.416152 18701 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.125:45865
I20260809 03:34:40.416389 18805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.125:45865 every 8 connection(s)
I20260809 03:34:40.417047 18701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
I20260809 03:34:40.419661 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:40.423234 18806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.424906 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18701
I20260809 03:34:40.425030 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-1/wal/instance
I20260809 03:34:40.425268 16249 external_mini_cluster.cc:1428] Setting key 2534031e858f89c10fa5ab707c0563ff
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.124@KRBTEST.COM" created.
I20260809 03:34:40.436862 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.424987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.15.222.126:41291)
0809 03:34:40.425137 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.425142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.425970 (+ 828us) server_negotiation.cc:407] Connection header received
0809 03:34:40.426883 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.426887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.426956 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.427050 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.428486 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.428997 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.430594 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.430821 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.430898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.433566 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.433569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.433571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.433595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.434973 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.435453 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.435455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.435456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.435532 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.435809 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.435811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.435812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.435985 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.436086 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.436573 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.436700 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:34:40.437577 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.425336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41291 (local address 127.0.0.1:33636)
0809 03:34:40.425804 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.425855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.426684 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.427242 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.427251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.427697 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.428330 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.428349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.429117 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.429120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.430482 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.430495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.430967 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.430972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.431173 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.431749 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.431768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.433383 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.435086 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.435091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.435102 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.435359 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.435596 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.435598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.435599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.435747 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.436065 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.436070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.436323 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.436499 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.436793 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260809 03:34:40.438995 18806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
I20260809 03:34:40.446965 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.439409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38614)
0809 03:34:40.439507 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.439520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.439585 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.440084 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.440086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.440246 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.440314 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.440319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.441200 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.441203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.441732 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.441741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.442267 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.442270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.442420 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.442832 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.442842 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.444085 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.445634 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.445637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.445638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.445852 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.446109 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.446111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.446113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.446183 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.446559 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.446561 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.446637 (+ 76us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.446714 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.446809 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":45}
I20260809 03:34:40.447023 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.439515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.15.222.125:45865)
0809 03:34:40.439777 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.439786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.439803 (+ 17us) server_negotiation.cc:407] Connection header received
0809 03:34:40.439832 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.439836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.439977 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.440089 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.440408 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.441096 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.441823 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.442171 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.442259 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.444218 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.444227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.444237 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.444258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.445540 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.445944 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.445950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.445951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.446038 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.446240 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.446241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.446242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.446476 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.446581 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.446772 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.446886 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260809 03:34:40.448629 18806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.449285 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.449040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:39964)
0809 03:34:40.449146 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.449226 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 03:34:40.449443 18728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
I20260809 03:34:40.450217 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.450018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:39972)
0809 03:34:40.450111 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.450162 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:34:40.450359 18806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
I20260809 03:34:40.450637 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:33139
--webserver_interface=127.15.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:34:40.459444 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } attempt: 3
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
I20260809 03:34:40.467918 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.459870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38618)
0809 03:34:40.459995 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.460009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.460089 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.460310 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.460312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.460459 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.460561 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.460566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.462022 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.462025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.462550 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.462558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.463105 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.463108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.463184 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.463614 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.463625 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.464978 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.466716 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.466719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.466720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.466899 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.467135 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.467137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.467138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.467204 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.467548 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.467550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.467629 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.467697 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.467797 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:34:40.468020 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.459940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.15.222.125:45865)
0809 03:34:40.460053 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.460057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.460073 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:34:40.460167 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.460170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.460216 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.460296 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.460652 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.461922 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.462643 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.463016 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.463110 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.465111 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.465115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.465117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.465144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.466619 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.466977 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.466980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.466982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.467062 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.467268 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.467270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.467272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.467471 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.467554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.467754 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.467864 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 03:34:40.468655 18689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.469311 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.469098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:39974)
0809 03:34:40.469207 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.469253 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 03:34:40.470178 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.469956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:39984)
0809 03:34:40.470085 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.470123 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260809 03:34:40.470364 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
I20260809 03:34:40.499584 18806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 1
I20260809 03:34:40.500357 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.500086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:39998)
0809 03:34:40.500222 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.500280 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 03:34:40.500979 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.500806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40000)
0809 03:34:40.500885 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.500925 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:34:40.501116 18806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
I20260809 03:34:40.501530 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 1
I20260809 03:34:40.502161 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.501928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40012)
0809 03:34:40.502051 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.502098 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 03:34:40.502789 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.502624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40024)
0809 03:34:40.502702 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.502740 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 03:34:40.502920 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
W20260809 03:34:40.562074 18815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:40.562337 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:40.562372 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:40.562440 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:40.566339 18815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:40.566402 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:40.566428 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:40.566458 18815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:40.566485 18815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:40.569304 18806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 2
I20260809 03:34:40.570101 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.569821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40036)
0809 03:34:40.569969 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.570027 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 03:34:40.570839 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.570637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40038)
0809 03:34:40.570735 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.570781 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:34:40.570957 18806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
I20260809 03:34:40.571862 18815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:41291,127.15.222.125:45865,127.15.222.124:33139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:33139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:40.572811 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:40.573796 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:34:40.578084 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 2
I20260809 03:34:40.578791 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.578517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40048)
0809 03:34:40.578652 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.578713 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:34:40.579392 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.579208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40052)
0809 03:34:40.579291 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.579334 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 03:34:40.579509 18689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:33139: Network error: Client connection negotiation failed: client connection to 127.15.222.124:33139: connect: Connection refused (error 111)
W20260809 03:34:40.582005 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.582075 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:40.582026 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:40.582326 18815 server_base.cc:1034] running on GCE node
I20260809 03:34:40.582919 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:40.583962 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:40.585111 18815 hybrid_clock.cc:648] HybridClock initialized: now 1786246480585094 us; error 34 us; skew 500 ppm
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:40.587740 18815 init.cc:377] Logged in from keytab as kudu/127.15.222.124@KRBTEST.COM (short username kudu)
I20260809 03:34:40.588549 18815 webserver.cc:460] Webserver started at http://127.15.222.124:43035/ using document root <none> and password file <none>
I20260809 03:34:40.589059 18815 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:40.589124 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:40.589318 18815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:40.591313 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data/instance:
uuid: "4fbdd19b99e04ab49e5269ddea4287d1"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "8ebf87fedf75e4e36cdba63e10f924e6"
server_key_iv: "74742d21cfa7472f5f30ba88726d3978"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.591856 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal/instance:
uuid: "4fbdd19b99e04ab49e5269ddea4287d1"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "8ebf87fedf75e4e36cdba63e10f924e6"
server_key_iv: "74742d21cfa7472f5f30ba88726d3978"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.595350 18815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 03:34:40.597584 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:40.598860 18815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 03:34:40.598977 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "4fbdd19b99e04ab49e5269ddea4287d1"
format_stamp: "Formatted at 2026-08-09 03:34:40 on dist-test-slave-cvwc"
server_key: "8ebf87fedf75e4e36cdba63e10f924e6"
server_key_iv: "74742d21cfa7472f5f30ba88726d3978"
server_key_version: "encryptionkey@0"
I20260809 03:34:40.599073 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:40.619166 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:40.622184 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:40.622354 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:40.628741 18815 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.124:33139
I20260809 03:34:40.628997 18925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.124:33139 every 8 connection(s)
I20260809 03:34:40.629647 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
I20260809 03:34:40.632266 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:40.635878 18928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.636353 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 18815
I20260809 03:34:40.636476 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786246200571663-16249-0/minicluster-data/master-2/wal/instance
I20260809 03:34:40.636721 16249 external_mini_cluster.cc:1428] Setting key a495add4f55fcec946f18c143ad30ecc
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:40.648296 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.637699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.15.222.126:41291)
0809 03:34:40.637828 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.637832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.638617 (+ 785us) server_negotiation.cc:407] Connection header received
0809 03:34:40.639606 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.639609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.639655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.639720 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.640579 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.641069 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.642138 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.642392 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.642468 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.644847 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.644850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.644854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.644893 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.646302 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.646802 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.646804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.646805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.646870 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.647189 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.647193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.647194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.647389 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.647475 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.648017 (+ 542us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.648136 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 03:34:40.649209 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.638755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.15.222.124:33139)
0809 03:34:40.638994 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.639005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.639044 (+ 39us) server_negotiation.cc:407] Connection header received
0809 03:34:40.639393 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.639416 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.639648 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.639812 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.640485 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.641433 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.642275 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.642679 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.642774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.645241 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.645248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.645261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.645287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.646917 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.647295 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.647299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.647300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.647367 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.647620 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.647623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.647624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.647892 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.648067 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.648239 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.648354 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"spinlock_wait_cycles":896,"thread_start_us":67,"threads_started":1}
I20260809 03:34:40.649206 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.638075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:41291 (local address 127.0.0.1:33650)
0809 03:34:40.638486 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.638524 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.639389 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.639789 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.639794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.640190 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.640433 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.640452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.641167 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.641170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.642033 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.642043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.642514 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.642518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.642687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.643210 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.643225 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.644684 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.646404 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.646411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.646421 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.646719 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.646951 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.646954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.646956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.647105 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.647480 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.647486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.647739 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.647955 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.648247 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260809 03:34:40.650645 18928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:40.658645 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.651161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:45865 (local address 127.0.0.1:38632)
0809 03:34:40.651378 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.651390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.651470 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.651721 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.651724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.651885 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.651980 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.651985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.652743 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.652746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.653322 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.653330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.653928 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.653931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.654026 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.654469 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.654479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.655567 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.657201 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.657204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.657207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.657414 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.657685 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.657689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.657691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.657807 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.658177 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.658180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.658278 (+ 98us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.658398 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.658536 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260809 03:34:40.658727 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.651267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.15.222.125:45865)
0809 03:34:40.651373 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.651377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.651431 (+ 54us) server_negotiation.cc:407] Connection header received
0809 03:34:40.651576 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.651580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.651630 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.651703 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.652088 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.652655 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.653443 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.653816 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.653899 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.655713 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.655716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.655719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.655752 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.657092 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.657509 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.657514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.657516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.657599 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.657904 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.657906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.657907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.658081 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.658177 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.658453 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.658579 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 03:34:40.659329 18928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:40.659863 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.651396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.15.222.126:41291)
0809 03:34:40.651524 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.651528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.651569 (+ 41us) server_negotiation.cc:407] Connection header received
0809 03:34:40.651687 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.651690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.651729 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.651800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.652369 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.652883 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.653782 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.654038 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.654132 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.656432 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.656436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.656439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.656464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.657690 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.658107 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.658110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.658112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.658191 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.658457 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.658460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.658462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.658680 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.659068 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.659650 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.659736 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 03:34:40.662097 18806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 3
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:40.667793 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.659764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40056)
0809 03:34:40.659905 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.659918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.659993 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.660233 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.660236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.660387 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.660460 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.660468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.661217 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.661220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.661832 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.661841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.662353 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.662356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.662435 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.662850 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.662860 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.664074 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.666363 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.666366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.666368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.666592 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.666856 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.666858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.666860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.666937 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.667337 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.667340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.667428 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.667526 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.667644 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 03:34:40.667954 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.659851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.15.222.124:33139)
0809 03:34:40.659949 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.659961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.659974 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:40.660079 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.660083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.660124 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.660183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.660579 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.661112 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.661926 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.662170 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.662258 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.664188 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.664191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.664194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.664216 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.666270 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.666694 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.666697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.666699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.666768 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.667017 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.667019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.667021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.667204 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.667306 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.667623 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.667759 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:34:40.670524 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.661089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.15.222.125:45865)
0809 03:34:40.661194 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.661197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.661209 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:40.661298 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.661300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.661339 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.661409 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.661973 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.662840 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.663844 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.664164 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.664268 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.666918 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.666922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.666925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.666952 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.668510 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.668973 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.668976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.668978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.669066 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.669397 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.669401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.669403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.669603 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.669882 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.670285 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.670385 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 03:34:40.671468 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.662434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40070)
0809 03:34:40.662536 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.662548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.662628 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.663192 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.663195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.663353 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.663426 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.663431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.664412 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.664415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.665081 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.665091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.665563 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.665567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.665650 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.666512 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.666524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.667833 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.669966 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.669968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.669970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.670151 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.670540 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.670543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.670545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.670632 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.671039 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.671042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.671124 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.671210 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.671327 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 03:34:40.671520 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.662588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.15.222.124:33139)
0809 03:34:40.662994 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.663005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.663018 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:40.663045 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.663047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.663081 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.663160 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.663536 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.664282 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.665192 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.665452 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.666042 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.668159 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.668161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.668163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.668183 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.669853 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.670338 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.670341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.670343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.670437 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.670737 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.670740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.670742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.670945 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.671030 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.671268 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.671364 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":135,"threads_started":1}
I20260809 03:34:40.671586 18928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: Bootstrap starting.
I20260809 03:34:40.674242 18928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:40.674382 18806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: Bootstrap starting.
I20260809 03:34:40.675145 18928 log.cc:824] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:40.675527 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:40.676590 18806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:40.677222 18928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: No bootstrap required, opened a new log
I20260809 03:34:40.677325 18806 log.cc:824] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:40.676887 18645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "553330ebe2274be3a54942691812bac5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666
I20260809 03:34:40.678644 18689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } attempt: 3
I20260809 03:34:40.679555 18806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: No bootstrap required, opened a new log
I20260809 03:34:40.681089 18928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.681383 18928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbdd19b99e04ab49e5269ddea4287d1, State: Initialized, Role: FOLLOWER
I20260809 03:34:40.681785 18928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.682369 18938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.682670 18938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.683197 18928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:40.683472 18806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.683671 18806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212c057ec49e4d2586a911f1b7ede14a, State: Initialized, Role: FOLLOWER
I20260809 03:34:40.683964 18806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
Aug 09 03:34:40 dist-test-slave-cvwc krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:40.685248 18806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: configured and running, proceeding with master startup.
W20260809 03:34:40.689215 18955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:40.689288 18955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:40.689595 18944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.689772 18944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.688956 18815 master_runner.cc:186] Error getting master registration for 127.15.222.126:41291: OK, instance_id { permanent_uuid: "553330ebe2274be3a54942691812bac5" instance_seqno: 1786246480219537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:34:40.691555 18815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 03:34:40.692106 18974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:40.692305 18974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:40.693356 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.679053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:33139 (local address 127.0.0.1:40076)
0809 03:34:40.679281 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:40.679298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:40.679384 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:40.679761 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:40.679764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:40.679958 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:40.680077 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.680086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.681200 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.681204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.682048 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:40.682062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.682772 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.682776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:40.682864 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.683350 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:40.683366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:40.684881 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:40.691646 (+ 6765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.691649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.691650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.691859 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.692198 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:40.692202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:40.692204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:40.692306 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:40.692786 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:40.692790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:40.692911 (+ 121us) client_negotiation.cc:769] Sending connection context
0809 03:34:40.693032 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 03:34:40.693185 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260809 03:34:40.693382 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.679132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.15.222.124:33139)
0809 03:34:40.679533 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.679538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.679551 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:40.679585 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.679588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.679628 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.680200 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.680311 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.681047 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.682198 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.682540 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.682647 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.685047 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.685052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.685055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.685081 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.687071 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.691976 (+ 4905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.691980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.691982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.692079 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.692422 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.692426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.692428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.692666 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.692785 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.693095 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.693219 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260809 03:34:40.692461 18701 master_runner.cc:186] Error getting master registration for 127.15.222.126:41291: OK, instance_id { permanent_uuid: "553330ebe2274be3a54942691812bac5" instance_seqno: 1786246480219537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:34:40.696261 18701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:34:40.696967 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: Bootstrap starting.
I20260809 03:34:40.699750 18689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:40.700641 18689 log.cc:824] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:40.703354 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: No bootstrap required, opened a new log
I20260809 03:34:40.706466 18689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.706686 18689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553330ebe2274be3a54942691812bac5, State: Initialized, Role: FOLLOWER
I20260809 03:34:40.707013 18689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.708231 18689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:40.708302 18645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 03:34:40.708428 18645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:40.709703 18645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.711367 18765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "553330ebe2274be3a54942691812bac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "212c057ec49e4d2586a911f1b7ede14a"
I20260809 03:34:40.711591 18765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:40.712910 18765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553330ebe2274be3a54942691812bac5 in term 1.
I20260809 03:34:40.714128 18609 leader_election.cc:302] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212c057ec49e4d2586a911f1b7ede14a, 553330ebe2274be3a54942691812bac5; no voters:
W20260809 03:34:40.716233 18613 tablet.cc:2369] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:40.717867 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.717998 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.718071 19005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:40.718140 19005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [term 1 LEADER]: Becoming Leader. State: Replica: 553330ebe2274be3a54942691812bac5, State: Running, Role: LEADER
I20260809 03:34:40.718464 19005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } }
I20260809 03:34:40.720654 18580 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260809 03:34:40.720762 18580 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:34:40.720945 18645 leader_election.cc:288] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [CANDIDATE]: Term 1 election: Requested vote from peers 212c057ec49e4d2586a911f1b7ede14a (127.15.222.125:45865), 4fbdd19b99e04ab49e5269ddea4287d1 (127.15.222.124:33139)
I20260809 03:34:40.721436 18892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "553330ebe2274be3a54942691812bac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fbdd19b99e04ab49e5269ddea4287d1"
I20260809 03:34:40.721665 18892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:40.722963 18892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553330ebe2274be3a54942691812bac5 in term 1.
W20260809 03:34:40.713287 19002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:40.725509 19002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:40.727389 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553330ebe2274be3a54942691812bac5. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.728006 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:40.737967 19019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:40.738945 19019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:40.743744 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.726031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.15.222.126:41291)
0809 03:34:40.726150 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.726155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.727984 (+ 1829us) server_negotiation.cc:407] Connection header received
0809 03:34:40.728290 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.728294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.728345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.728398 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.729441 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.730236 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.731435 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.731766 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.731844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.733802 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.733807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.733809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.733838 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.740782 (+ 6944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.741316 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.741320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.741322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.741415 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.741789 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.741792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.741795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.742022 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.742127 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.743455 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.743575 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:40.749305 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.726735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.15.222.125:45865)
0809 03:34:40.726849 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.726854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.733970 (+ 7116us) server_negotiation.cc:407] Connection header received
0809 03:34:40.734168 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.734172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.734227 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.734285 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.735029 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.735714 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.736879 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.737199 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.737289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.745387 (+ 8098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.745391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.745394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.745423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.747163 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.747787 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.747791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.747793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.747892 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.748242 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.748246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.748248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.748473 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.748595 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.749000 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.749118 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 03:34:40.752274 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:40.727154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.15.222.124:33139)
0809 03:34:40.727259 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.727263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.735208 (+ 7945us) server_negotiation.cc:407] Connection header received
0809 03:34:40.735391 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.735395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.735442 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.735495 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.736405 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.737112 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.746247 (+ 9135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:40.746584 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:40.746672 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:40.748368 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:40.748372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:40.748375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:40.748402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:40.750110 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.750608 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.750612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.750614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.750732 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:40.751064 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:40.751068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:40.751070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:40.751279 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:40.751419 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:40.751979 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 03:34:40.752095 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 03:34:40.757131 18765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [term 1 FOLLOWER]: Refusing update from remote peer 553330ebe2274be3a54942691812bac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:40.757237 18892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [term 1 FOLLOWER]: Refusing update from remote peer 553330ebe2274be3a54942691812bac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:40.757784 18981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:40.762475 18944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553330ebe2274be3a54942691812bac5. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.762581 18944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.762735 18981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:40.766323 18938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553330ebe2274be3a54942691812bac5. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.766436 18938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.769003 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.769121 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:40.769399 19021 mvcc.cc:205] Tried to move back new op lower bound from 7316465585174134784 to 7316465585053966336. Current Snapshot: MvccSnapshot[applied={T|T < 7316465585174134784}]
I20260809 03:34:40.769565 18944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.769665 18944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.770579 19022 mvcc.cc:205] Tried to move back new op lower bound from 7316465585174134784 to 7316465585053966336. Current Snapshot: MvccSnapshot[applied={T|T < 7316465585174134784}]
I20260809 03:34:40.775758 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.775872 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:40.776963 18938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "553330ebe2274be3a54942691812bac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553330ebe2274be3a54942691812bac5" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 41291 } } peers { permanent_uuid: "212c057ec49e4d2586a911f1b7ede14a" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 45865 } } peers { permanent_uuid: "4fbdd19b99e04ab49e5269ddea4287d1" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 33139 } } }
I20260809 03:34:40.777076 18938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:40.777845 19019 catalog_manager.cc:1347] Generated new cluster ID: 397bb885b81c46d09ca104abae0a5c52
I20260809 03:34:40.777906 19019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:40.792114 19019 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:40.793355 19019 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:40.805871 19025 mvcc.cc:205] Tried to move back new op lower bound from 7316465585174134784 to 7316465585053966336. Current Snapshot: MvccSnapshot[applied={T|T < 7316465585174134784}]
I20260809 03:34:40.811233 19019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 553330ebe2274be3a54942691812bac5: Generated new TSK 0
I20260809 03:34:40.812016 19019 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:34:40.844411 18697 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.841418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.15.222.126:41291)
0809 03:34:40.841549 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.841554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.843295 (+ 1741us) server_negotiation.cc:407] Connection header received
0809 03:34:40.843505 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.843509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.843561 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.843616 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.844273 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 03:34:40.845916 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.842260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.15.222.125:45865)
0809 03:34:40.842386 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.842391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.844924 (+ 2533us) server_negotiation.cc:407] Connection header received
0809 03:34:40.845106 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.845110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.845162 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.845219 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.845789 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38650: BlockingRecv error: recv got EOF from 127.0.0.1:38650 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 03:34:40.847373 18935 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.842625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.15.222.124:33139)
0809 03:34:40.842752 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.842757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.846393 (+ 3636us) server_negotiation.cc:407] Connection header received
0809 03:34:40.846584 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.846588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.846642 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.846698 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.847250 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40108: BlockingRecv error: recv got EOF from 127.0.0.1:40108 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 03:34:40.857542 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.855923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.15.222.125:45865)
0809 03:34:40.856034 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.856038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.856560 (+ 522us) server_negotiation.cc:407] Connection header received
0809 03:34:40.856780 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.856784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.856857 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.856912 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.857434 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38652: BlockingRecv error: recv got EOF from 127.0.0.1:38652 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 03:34:40.862799 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.861138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.15.222.125:45865)
0809 03:34:40.861269 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.861274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.861860 (+ 586us) server_negotiation.cc:407] Connection header received
0809 03:34:40.862067 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.862071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.862122 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.862174 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.862702 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38666: BlockingRecv error: recv got EOF from 127.0.0.1:38666 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 03:34:40.868440 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.866831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.15.222.125:45865)
0809 03:34:40.866933 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.866938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.867512 (+ 574us) server_negotiation.cc:407] Connection header received
0809 03:34:40.867707 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.867711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.867763 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.867817 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.868344 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38682: BlockingRecv error: recv got EOF from 127.0.0.1:38682 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 03:34:40.876318 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.874598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.15.222.125:45865)
0809 03:34:40.874739 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.874744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.875322 (+ 578us) server_negotiation.cc:407] Connection header received
0809 03:34:40.875522 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.875525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.875578 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.875631 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.876177 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38690: BlockingRecv error: recv got EOF from 127.0.0.1:38690 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 03:34:40.889142 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.887126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.15.222.125:45865)
0809 03:34:40.887273 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.887278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.887969 (+ 691us) server_negotiation.cc:407] Connection header received
0809 03:34:40.888238 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.888242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.888310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.888375 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.889015 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38702: BlockingRecv error: recv got EOF from 127.0.0.1:38702 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 03:34:40.909534 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.907694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.15.222.125:45865)
0809 03:34:40.907822 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.907827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.908462 (+ 635us) server_negotiation.cc:407] Connection header received
0809 03:34:40.908675 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.908679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.908737 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.908794 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.909407 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38716: BlockingRecv error: recv got EOF from 127.0.0.1:38716 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 03:34:40.945223 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:40.944150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.15.222.125:45865)
0809 03:34:40.944327 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:40.944331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:40.944434 (+ 103us) server_negotiation.cc:407] Connection header received
0809 03:34:40.944562 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:40.944565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:40.944615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:40.944689 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:40.945093 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38728: BlockingRecv error: recv got EOF from 127.0.0.1:38728 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 03:34:41.012578 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:41.011531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.15.222.125:45865)
0809 03:34:41.011658 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.011663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.011768 (+ 105us) server_negotiation.cc:407] Connection header received
0809 03:34:41.011895 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.011898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.011956 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.012056 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:41.012456 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38732: BlockingRecv error: recv got EOF from 127.0.0.1:38732 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 03:34:41.143565 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:41.142533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.15.222.125:45865)
0809 03:34:41.142700 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.142703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.142797 (+ 94us) server_negotiation.cc:407] Connection header received
0809 03:34:41.142931 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.142933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.142983 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.143053 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:41.143442 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38748: BlockingRecv error: recv got EOF from 127.0.0.1:38748 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 03:34:41.402738 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:41.401584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.15.222.125:45865)
0809 03:34:41.401745 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.401750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.401831 (+ 81us) server_negotiation.cc:407] Connection header received
0809 03:34:41.401991 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.401994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.402044 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.402112 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:41.402617 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 03:34:41.690625 18955 catalog_manager.cc:1259] Loaded cluster ID: 397bb885b81c46d09ca104abae0a5c52
I20260809 03:34:41.690709 18955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: loading cluster ID for follower catalog manager: success
I20260809 03:34:41.692401 18955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: acquiring CA information for follower catalog manager: success
I20260809 03:34:41.693490 18974 catalog_manager.cc:1259] Loaded cluster ID: 397bb885b81c46d09ca104abae0a5c52
I20260809 03:34:41.693548 18974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: loading cluster ID for follower catalog manager: success
I20260809 03:34:41.693576 18955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fbdd19b99e04ab49e5269ddea4287d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:34:41.695067 18974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: acquiring CA information for follower catalog manager: success
I20260809 03:34:41.696059 18974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 212c057ec49e4d2586a911f1b7ede14a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:34:41.920132 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:41.916678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.15.222.125:45865)
0809 03:34:41.916931 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.916935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.916951 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:34:41.917081 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.917083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.917212 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.917330 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:41.917538 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.918074 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.918875 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.919084 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.919151 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:41.919385 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:41.919822 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:41.919911 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:41.919957 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 03:34:41.920006 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260809 03:34:41.926578 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:41.923113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.15.222.124:33139)
0809 03:34:41.923335 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.923338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.923351 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:41.923465 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.923467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.923579 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.923671 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:41.923889 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.924425 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.925244 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.925461 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.925527 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:41.925808 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:41.926221 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:41.926312 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:41.926364 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 03:34:41.926423 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260809 03:34:41.933183 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:41.929406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.15.222.124:33139)
0809 03:34:41.929523 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.929528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.929961 (+ 433us) server_negotiation.cc:407] Connection header received
0809 03:34:41.930009 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.930012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.930059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.930125 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:41.930490 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.930963 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.932029 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.932233 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.932315 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:41.932622 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:41.932695 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:41.932755 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:41.933017 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 03:34:41.933065 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:41.934000 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:41.929295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.15.222.126:41291)
0809 03:34:41.929614 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.929618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.929633 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:34:41.929851 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.929855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.930001 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.930354 (+ 353us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:41.930470 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.931232 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.932313 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.932565 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.932658 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:41.932939 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:41.933264 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:41.933397 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:41.933463 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 03:34:41.933878 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":89,"threads_started":1}
I20260809 03:34:41.934520 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:41.929570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.15.222.125:45865)
0809 03:34:41.929690 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:41.929695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:41.930060 (+ 365us) server_negotiation.cc:407] Connection header received
0809 03:34:41.930141 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:41.930144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:41.930192 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:41.930262 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:41.930458 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.931227 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.933323 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:41.933616 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:41.933688 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:41.934075 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:41.934154 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:41.934227 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:41.934353 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 03:34:41.934404 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:41.937122 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18580
I20260809 03:34:41.953071 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18701
I20260809 03:34:41.964978 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 18815
2026-08-09T03:34:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3153 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 03:34:42 dist-test-slave-cvwc krb5kdc[19166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 03:34:42 dist-test-slave-cvwc krb5kdc[19166](info): set up 2 sockets
Aug 09 03:34:42 dist-test-slave-cvwc krb5kdc[19166](info): commencing operation
krb5kdc: starting...
W20260809 03:34:43.096568 16249 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/test-user.keytab.
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T03:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T03:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
Entry for principal kudu/127.15.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab.
I20260809 03:34:43.212560 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:42075
--webserver_interface=127.15.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 03:34:43.322456 19188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:43.322708 19188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:43.322731 19188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:43.322790 19188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 03:34:43.326648 19188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:43.326699 19188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:43.326716 19188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:43.326732 19188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:43.326759 19188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:43.331758 19188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.126.keytab
--principal=kudu/127.15.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.126:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:43.332674 19188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:43.333618 19188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:34:43.341590 19188 server_base.cc:1034] running on GCE node
W20260809 03:34:43.341612 19199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.341607 19200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.341811 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:43.342351 19188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:43.343354 19188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:43.344507 19188 hybrid_clock.cc:648] HybridClock initialized: now 1786246483344491 us; error 28 us; skew 500 ppm
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:43.347105 19188 init.cc:377] Logged in from keytab as kudu/127.15.222.126@KRBTEST.COM (short username kudu)
I20260809 03:34:43.347944 19188 webserver.cc:460] Webserver started at http://127.15.222.126:42601/ using document root <none> and password file <none>
I20260809 03:34:43.348449 19188 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:43.348515 19188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:43.348696 19188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:43.350674 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data/instance:
uuid: "135ea63f72ee40c8b25ad7596f42b387"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "d7a1661d48c5039e7bfdc309e5f14789"
server_key_iv: "0c44218c97088a9e00ff7da3fd45d0a8"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.351205 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance:
uuid: "135ea63f72ee40c8b25ad7596f42b387"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "d7a1661d48c5039e7bfdc309e5f14789"
server_key_iv: "0c44218c97088a9e00ff7da3fd45d0a8"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.354676 19188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:43.356869 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:43.358072 19188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:43.358191 19188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal
uuid: "135ea63f72ee40c8b25ad7596f42b387"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "d7a1661d48c5039e7bfdc309e5f14789"
server_key_iv: "0c44218c97088a9e00ff7da3fd45d0a8"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.358299 19188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:43.367784 19188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:43.370710 19188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:43.370874 19188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:43.377460 19188 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.126:42075
I20260809 03:34:43.377782 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.126:42075 every 8 connection(s)
I20260809 03:34:43.378417 19188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/data/info.pb
I20260809 03:34:43.381031 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:43.384689 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.387742 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19188
I20260809 03:34:43.387840 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-0/wal/instance
I20260809 03:34:43.388114 16249 external_mini_cluster.cc:1428] Setting key fd8b4c3762ef29b451d7e923cfdb6da3
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.125@KRBTEST.COM" created.
I20260809 03:34:43.399561 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.386761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.15.222.126:42075)
0809 03:34:43.387188 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.387203 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.387380 (+ 177us) server_negotiation.cc:407] Connection header received
0809 03:34:43.388472 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.388478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.388717 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.388947 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.389661 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.390873 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.391824 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.392350 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.392510 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.394719 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.394726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.394740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.394767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.396560 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.397009 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.397016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.397017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.397160 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.397447 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.397450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.397451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.397915 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.398044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.398439 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.398731 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":305,"spinlock_wait_cycles":32640,"thread_start_us":152,"threads_started":1}
I20260809 03:34:43.399564 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.386760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42075 (local address 127.0.0.1:50802)
0809 03:34:43.387222 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.387271 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.388155 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.388909 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.388914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.389231 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.389506 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.389519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.390997 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.390999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.391707 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.391716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.392499 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.392502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.392580 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.393142 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.393163 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.394568 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.396662 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.396670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.396677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.396907 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.397249 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.397251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.397252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.397361 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.398019 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.398026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.398186 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.398339 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.398650 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":24,"thread_start_us":124,"threads_started":1}
I20260809 03:34:43.402347 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.403067 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.402829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57198)
0809 03:34:43.402946 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.403013 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 03:34:43.403244 19218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
I20260809 03:34:43.404018 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.403811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57206)
0809 03:34:43.403911 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.403957 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:34:43.404168 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:43281: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
Entry for principal kudu/127.15.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab.
I20260809 03:34:43.414058 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:43281
--webserver_interface=127.15.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:34:43.453377 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } attempt: 1
I20260809 03:34:43.454304 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.454005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57222)
0809 03:34:43.454158 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.454223 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 03:34:43.455004 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.454813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57230)
0809 03:34:43.454904 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.454947 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 03:34:43.455160 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:43281: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
I20260809 03:34:43.523304 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } attempt: 2
W20260809 03:34:43.523360 19315 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:43.523692 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:43.523737 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:43.523790 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 03:34:43.524127 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.523837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57240)
0809 03:34:43.523987 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.524050 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 03:34:43.524746 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.524564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57242)
0809 03:34:43.524646 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.524689 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 03:34:43.524873 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.125:43281: Network error: Client connection negotiation failed: client connection to 127.15.222.125:43281: connect: Connection refused (error 111)
W20260809 03:34:43.528024 19315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:43.528074 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:43.528095 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:43.528124 19315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:43.528153 19315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:43.533236 19315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.125.keytab
--principal=kudu/127.15.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.125:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:43.534238 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:43.535199 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 03:34:43.543184 19324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.543207 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.543184 19323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:43.543416 19315 server_base.cc:1034] running on GCE node
I20260809 03:34:43.543989 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:43.544988 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:43.546140 19315 hybrid_clock.cc:648] HybridClock initialized: now 1786246483546114 us; error 36 us; skew 500 ppm
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:43.548677 19315 init.cc:377] Logged in from keytab as kudu/127.15.222.125@KRBTEST.COM (short username kudu)
I20260809 03:34:43.549484 19315 webserver.cc:460] Webserver started at http://127.15.222.125:40549/ using document root <none> and password file <none>
I20260809 03:34:43.550027 19315 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:43.550089 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:43.550256 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:43.552150 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data/instance:
uuid: "93a12e8ba6874e64a559d48df27dfc77"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "c4393c4b0c176a8b2d080c391f9c9ce9"
server_key_iv: "4b60fa8386c355cbd9b0e41c153ed01d"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.552667 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal/instance:
uuid: "93a12e8ba6874e64a559d48df27dfc77"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "c4393c4b0c176a8b2d080c391f9c9ce9"
server_key_iv: "4b60fa8386c355cbd9b0e41c153ed01d"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.556170 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 03:34:43.558413 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:43.559545 19315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 03:34:43.559643 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal
uuid: "93a12e8ba6874e64a559d48df27dfc77"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "c4393c4b0c176a8b2d080c391f9c9ce9"
server_key_iv: "4b60fa8386c355cbd9b0e41c153ed01d"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.559721 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:43.583662 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:43.586305 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:43.586455 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:43.592859 19315 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.125:43281
I20260809 03:34:43.593113 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.125:43281 every 8 connection(s)
I20260809 03:34:43.593815 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/data/info.pb
I20260809 03:34:43.596520 19434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:43.599799 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19315
I20260809 03:34:43.599934 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-1/wal/instance
I20260809 03:34:43.600236 16249 external_mini_cluster.cc:1428] Setting key ee131661263d40a10722261335b6b6c3
I20260809 03:34:43.600376 19434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.222.124@KRBTEST.COM" created.
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:43.615339 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.602519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.15.222.126:42075)
0809 03:34:43.602636 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.602640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.603597 (+ 957us) server_negotiation.cc:407] Connection header received
0809 03:34:43.604489 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.604492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.604557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.604630 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.605828 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.606339 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.607430 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.607661 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.607752 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.611794 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.611798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.611801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.611830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.613259 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.613832 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.613835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.613836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.613916 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.614251 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.614253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.614254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.614425 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.614494 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.615114 (+ 620us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.615214 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:34:43.616035 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } attempt: 3
I20260809 03:34:43.616429 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.602992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42075 (local address 127.0.0.1:50806)
0809 03:34:43.603455 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.603498 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.604284 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.604801 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.604807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.605209 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.605665 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.605678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.606463 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.606466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.607303 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.607313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.607802 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.607806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.607969 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.608483 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.608506 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.611623 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.613375 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.613390 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.613404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.613750 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.613995 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.613998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.614000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.614173 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.614531 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.614538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.614852 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.615052 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.615376 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260809 03:34:43.618088 19434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
Entry for principal kudu/127.15.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab.
I20260809 03:34:43.626765 16249 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
/tmp/dist-test-taskbUfAuu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:35595
--webserver_interface=127.15.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.222.84:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 03:34:43.627106 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.616417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57246)
0809 03:34:43.616975 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.616990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.617061 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.617563 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.617566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.617755 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.618265 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.618276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.619467 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.619469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.620282 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.620293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.620842 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.620846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.620964 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.621373 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.621388 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.623043 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.625527 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.625531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.625533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.625743 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.626038 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.626041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.626043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.626157 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.626663 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.626665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.626748 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.626824 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.626953 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480}
I20260809 03:34:43.627238 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.616619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.15.222.125:43281)
0809 03:34:43.617094 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.617105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.617128 (+ 23us) server_negotiation.cc:407] Connection header received
0809 03:34:43.617169 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.617174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.617368 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.617578 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.618377 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.619341 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.620384 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.620735 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.620829 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.623167 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.623177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.623190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.623212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.625431 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.625841 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.625847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.625849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.625952 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.626267 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.626270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.626272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.626569 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.626705 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.626958 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.627088 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":141,"threads_started":1}
I20260809 03:34:43.627450 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.618555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57252)
0809 03:34:43.618674 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.618689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.618758 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.619195 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.619198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.619346 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.619446 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.619452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.620463 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.620466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.621013 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.621022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.621424 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.621427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.621506 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.621911 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.621924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.623468 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.625541 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.625544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.625546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.625798 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.626081 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.626083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.626085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.626160 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.627044 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.627046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.627116 (+ 70us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.627186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.627297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 03:34:43.627904 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.618725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.15.222.125:43281)
0809 03:34:43.618987 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.618994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.619005 (+ 11us) server_negotiation.cc:407] Connection header received
0809 03:34:43.619034 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.619037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.619076 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.619164 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.619546 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.620364 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.621110 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.621330 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.621412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.623652 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.623655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.623658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.623683 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.625452 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.625907 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.625910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.625911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.625987 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.626742 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.626745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.626747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.626951 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.627583 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.627651 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.627744 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"spinlock_wait_cycles":9088,"thread_start_us":98,"threads_started":1}
I20260809 03:34:43.629132 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.629233 19434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.629848 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.629592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57438)
0809 03:34:43.629690 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.629786 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 03:34:43.629942 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.629621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57432)
0809 03:34:43.629818 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.629867 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260809 03:34:43.630077 19343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
I20260809 03:34:43.630692 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.630421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57454)
0809 03:34:43.630581 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.630626 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260809 03:34:43.630906 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.630648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57470)
0809 03:34:43.630822 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.630861 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260809 03:34:43.630895 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
W20260809 03:34:43.631042 19434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
I20260809 03:34:43.662053 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 1
I20260809 03:34:43.662835 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.662538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57476)
0809 03:34:43.662696 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.662754 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 03:34:43.663609 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.663395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57486)
0809 03:34:43.663503 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.663550 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 03:34:43.663757 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
I20260809 03:34:43.680261 19434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 1
I20260809 03:34:43.681023 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.680760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57496)
0809 03:34:43.680891 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.680949 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 03:34:43.681664 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.681461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57508)
0809 03:34:43.681564 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.681608 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 03:34:43.681874 19434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
W20260809 03:34:43.737365 19444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 03:34:43.737604 19444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 03:34:43.737625 19444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 03:34:43.737679 19444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 03:34:43.738921 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 2
I20260809 03:34:43.739701 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.739404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57512)
0809 03:34:43.739565 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.739620 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 03:34:43.740393 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.740191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57524)
0809 03:34:43.740291 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.740333 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 03:34:43.740531 19304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
W20260809 03:34:43.741992 19444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 03:34:43.742041 19444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 03:34:43.742061 19444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 03:34:43.742090 19444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 03:34:43.742120 19444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 03:34:43.747157 19444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.222.84:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.222.126:42075,127.15.222.125:43281,127.15.222.124:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/krb5kdc/kudu_127.15.222.124.keytab
--principal=kudu/127.15.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.222.124:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 02:17:35 UTC on ccycloud.quasar-qimiwd.root.comops.site
build id 4575723
I20260809 03:34:43.748070 19444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 03:34:43.749032 19444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 03:34:43.750042 19434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 2
I20260809 03:34:43.750666 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.750436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57526)
0809 03:34:43.750553 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.750601 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 03:34:43.751322 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.751136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57532)
0809 03:34:43.751225 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.751268 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 03:34:43.751447 19434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.222.124:35595: Network error: Client connection negotiation failed: client connection to 127.15.222.124:35595: connect: Connection refused (error 111)
W20260809 03:34:43.757218 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.757223 19451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 03:34:43.757229 19452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 03:34:43.757490 19444 server_base.cc:1034] running on GCE node
I20260809 03:34:43.758046 19444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 03:34:43.759061 19444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 03:34:43.760216 19444 hybrid_clock.cc:648] HybridClock initialized: now 1786246483760199 us; error 30 us; skew 500 ppm
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 03:34:43.762876 19444 init.cc:377] Logged in from keytab as kudu/127.15.222.124@KRBTEST.COM (short username kudu)
I20260809 03:34:43.763721 19444 webserver.cc:460] Webserver started at http://127.15.222.124:36437/ using document root <none> and password file <none>
I20260809 03:34:43.764246 19444 fs_manager.cc:359] Metadata directory not provided
I20260809 03:34:43.764321 19444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 03:34:43.764530 19444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 03:34:43.766544 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data/instance:
uuid: "e118d8dabf114541af326f7215571595"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "852bc06912d42b61656a09ff2db96c24"
server_key_iv: "18e8b634fdab012f008e522dc0b53dc1"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.767066 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal/instance:
uuid: "e118d8dabf114541af326f7215571595"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "852bc06912d42b61656a09ff2db96c24"
server_key_iv: "18e8b634fdab012f008e522dc0b53dc1"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.770524 19444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 03:34:43.772715 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 03:34:43.773903 19444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 03:34:43.774025 19444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data,/tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal
uuid: "e118d8dabf114541af326f7215571595"
format_stamp: "Formatted at 2026-08-09 03:34:43 on dist-test-slave-cvwc"
server_key: "852bc06912d42b61656a09ff2db96c24"
server_key_iv: "18e8b634fdab012f008e522dc0b53dc1"
server_key_version: "encryptionkey@0"
I20260809 03:34:43.774113 19444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 03:34:43.785356 19444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 03:34:43.788376 19444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 03:34:43.788552 19444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 03:34:43.795222 19444 rpc_server.cc:307] RPC server started. Bound to: 127.15.222.124:35595
I20260809 03:34:43.795485 19551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.222.124:35595 every 8 connection(s)
I20260809 03:34:43.796154 19444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/data/info.pb
I20260809 03:34:43.798838 19555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 03:34:43.802467 19555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.803714 16249 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu as pid 19444
I20260809 03:34:43.803839 16249 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbUfAuu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786246200571663-16249-0/minicluster-data/master-2/wal/instance
I20260809 03:34:43.804162 16249 external_mini_cluster.cc:1428] Setting key af01ea4338fe014b4f4023d50793460e
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:43.815128 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.804282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.15.222.126:42075)
0809 03:34:43.804425 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.804430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.805352 (+ 922us) server_negotiation.cc:407] Connection header received
0809 03:34:43.806259 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.806261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.806310 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.806373 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.807270 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.807760 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.808791 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.809056 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.809131 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.811567 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.811571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.811574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.811601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.813048 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.813657 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.813660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.813661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.813763 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.814084 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.814088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.814089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.814258 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.814341 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.814848 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.814970 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 03:34:43.815902 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.804793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.15.222.124:35595)
0809 03:34:43.805218 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.805235 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.805272 (+ 37us) server_negotiation.cc:407] Connection header received
0809 03:34:43.806051 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.806075 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.806312 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.806482 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.807152 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.808102 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.808927 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.809345 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.809438 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.811989 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.811994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.812003 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.812023 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.813573 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.814069 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.814075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.814077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.814152 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.814402 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.814405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.814407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.814639 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.814818 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.815032 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.815158 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"spinlock_wait_cycles":25472,"thread_start_us":157,"threads_started":1}
I20260809 03:34:43.815908 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.804733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.126:42075 (local address 127.0.0.1:50822)
0809 03:34:43.805198 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.805247 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.806051 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.806446 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.806452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.806884 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.807125 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.807144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.807859 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.807862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.808680 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.808689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.809180 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.809183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.809338 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.809916 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.809932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.811419 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.813153 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.813161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.813171 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.813574 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.813849 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.813852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.813854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.813995 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.814354 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.814361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.814601 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.814769 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.815066 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"spinlock_wait_cycles":1920,"thread_start_us":130,"threads_started":1}
I20260809 03:34:43.817677 19555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.126@KRBTEST.COM
I20260809 03:34:43.826113 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.818121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.125:43281 (local address 127.0.0.1:57262)
0809 03:34:43.818232 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.818245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.818312 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.818710 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.818714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.818859 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.818967 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.818974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.819940 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.819943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.820707 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.820715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.821259 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.821262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.821353 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.821878 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.821902 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.823123 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.824758 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.824762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.824763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.824977 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.825199 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.825202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.825204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.825289 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.825666 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.825669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.825782 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.825879 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.825979 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 03:34:43.826169 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.818372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.15.222.125:43281)
0809 03:34:43.818511 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.818515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.818528 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:43.818563 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.818565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.818603 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.818660 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.819076 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.819849 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.820824 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.821172 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.821256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.823246 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.823250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.823251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.823273 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.824661 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.825062 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.825064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.825065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.825129 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.825365 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.825368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.825369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.825578 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.825674 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.825936 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.826051 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:34:43.826769 19555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } has no permanent_uuid. Determining permanent_uuid...
I20260809 03:34:43.827039 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.818025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.15.222.126:42075)
0809 03:34:43.818127 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.818131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.818169 (+ 38us) server_negotiation.cc:407] Connection header received
0809 03:34:43.818292 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.818295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.818334 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.818403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.819043 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.819777 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.820824 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.821159 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.821252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.823766 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.823769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.823772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.823795 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.825249 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.825673 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.825676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.825678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.825783 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.826053 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.826056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.826058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.826257 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.826445 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.826783 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.826904 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.124@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.222.125@KRBTEST.COM
I20260809 03:34:43.833881 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.827178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57556)
0809 03:34:43.827275 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.827287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.827352 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.827632 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.827634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.827785 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.827851 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.827857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.828647 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.828650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.829123 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.829129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.829530 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.829533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.829644 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.830054 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.830067 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.831181 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.832576 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.832579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.832581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.832779 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.833051 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.833053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.833055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.833126 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.833498 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.833501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.833572 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.833646 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.833762 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":14}
I20260809 03:34:43.834017 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.827356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.15.222.124:35595)
0809 03:34:43.827446 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.827449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.827461 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:43.827491 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.827494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.827530 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.827597 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.827941 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.828559 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.829216 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.829445 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.829524 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.831292 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.831295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.831298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.831320 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.832485 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.832875 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.832877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.832879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.832959 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.833204 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.833207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.833208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.833402 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.833494 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.833738 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.833838 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":37}
I20260809 03:34:43.835701 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.828115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.15.222.125:43281)
0809 03:34:43.828212 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.828216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.828228 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:43.828277 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.828280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.828315 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.828383 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.828862 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.829299 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.830050 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.830256 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.830335 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.832512 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.832515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.832517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.832540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.834247 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.834612 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.834615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.834616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.834683 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.834928 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.834931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.834933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.835083 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.835255 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.835390 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.835484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 03:34:43.837225 19555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: Bootstrap starting.
I20260809 03:34:43.838754 16249 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 03:34:43.839675 19304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 3
I20260809 03:34:43.839740 19555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:43.840602 19555 log.cc:824] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:43.839779 19267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "135ea63f72ee40c8b25ad7596f42b387"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836
I20260809 03:34:43.842545 19555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: No bootstrap required, opened a new log
I20260809 03:34:43.842589 19434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } attempt: 3
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.126@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:43.847122 19555 raft_consensus.cc:348] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.847388 19555 raft_consensus.cc:729] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e118d8dabf114541af326f7215571595, State: Initialized, Role: FOLLOWER
Aug 09 03:34:43 dist-test-slave-cvwc krb5kdc[19166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786246483, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.222.125@KRBTEST.COM for kudu/127.15.222.124@KRBTEST.COM
I20260809 03:34:43.847726 19555 consensus_queue.cc:260] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.848271 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.848554 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.849102 19555 sys_catalog.cc:564] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:43.849216 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.840027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57558)
0809 03:34:43.840147 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.840160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.840229 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.840468 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.840471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.840634 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.840718 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.840724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.841522 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.841526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.842363 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.842372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.842898 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.842901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.842984 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.843461 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.843478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.844784 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.847653 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.847656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.847658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.847874 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.848175 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.848178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.848180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.848269 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.848736 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.848740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.848837 (+ 97us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.848934 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.849056 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 03:34:43.849828 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.840191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.15.222.124:35595)
0809 03:34:43.840281 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.840285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.840297 (+ 12us) server_negotiation.cc:407] Connection header received
0809 03:34:43.840326 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.840328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.840363 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.840450 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.840842 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.841410 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.842484 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.842794 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.842881 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.844948 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.844953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.844956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.844980 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.847537 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.847990 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.847994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.847995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.848073 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.848385 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.848388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.848390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.848621 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.849486 (+ 865us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.849558 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.849646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 03:34:43.851285 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.842934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.222.124:35595 (local address 127.0.0.1:57562)
0809 03:34:43.843025 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 03:34:43.843038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 03:34:43.843106 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 03:34:43.843674 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 03:34:43.843677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 03:34:43.843844 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0809 03:34:43.843930 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.843937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.844929 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.844932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.845496 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 03:34:43.845504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.846159 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.846162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 03:34:43.846247 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.846619 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 03:34:43.846633 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 03:34:43.847835 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 03:34:43.849695 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.849698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.849700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.849992 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.850354 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 03:34:43.850357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 03:34:43.850359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 03:34:43.850455 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 03:34:43.850815 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 03:34:43.850818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 03:34:43.850920 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 03:34:43.851017 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 03:34:43.851138 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 03:34:43.851323 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.843149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.15.222.124:35595)
0809 03:34:43.843461 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.843469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.843482 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:43.843514 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.843517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.843555 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.843643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.844070 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.844806 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.845600 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.845972 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.846095 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.848002 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.848006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.848008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.848027 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.849595 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.850187 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.850190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.850192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.850263 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.850529 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.850532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.850533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.850736 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.850821 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.851075 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.851168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":75,"threads_started":1}
I20260809 03:34:43.852633 19304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: Bootstrap starting.
I20260809 03:34:43.853516 19434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: Bootstrap starting.
I20260809 03:34:43.855186 19304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:43.855865 19434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: Neither blocks nor log segments found. Creating new log.
I20260809 03:34:43.856071 19304 log.cc:824] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: Log is configured to *not* fsync() on all Append() calls
I20260809 03:34:43.856865 19434 log.cc:824] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: Log is configured to *not* fsync() on all Append() calls
W20260809 03:34:43.858283 19579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:43.858398 19579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:43.858698 19304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: No bootstrap required, opened a new log
I20260809 03:34:43.860141 19434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: No bootstrap required, opened a new log
I20260809 03:34:43.857816 19444 master_runner.cc:186] Error getting master registration for 127.15.222.126:42075: OK, instance_id { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" instance_seqno: 1786246483375614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:34:43.860911 19444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:34:43.861610 19304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.861855 19304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135ea63f72ee40c8b25ad7596f42b387, State: Initialized, Role: FOLLOWER
I20260809 03:34:43.862174 19304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.863509 19304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:43.863588 19267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 03:34:43.863704 19267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:43.864173 19434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.864372 19434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a12e8ba6874e64a559d48df27dfc77, State: Initialized, Role: FOLLOWER
I20260809 03:34:43.864691 19434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.864898 19267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.865988 19434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 03:34:43.866067 19267 leader_election.cc:288] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [CANDIDATE]: Term 1 election: Requested vote from peers 93a12e8ba6874e64a559d48df27dfc77 (127.15.222.125:43281), e118d8dabf114541af326f7215571595 (127.15.222.124:35595)
I20260809 03:34:43.866856 19512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "135ea63f72ee40c8b25ad7596f42b387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e118d8dabf114541af326f7215571595"
I20260809 03:34:43.867091 19512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:43.867127 19386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "135ea63f72ee40c8b25ad7596f42b387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "93a12e8ba6874e64a559d48df27dfc77"
I20260809 03:34:43.867324 19386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [term 0 FOLLOWER]: Advancing to term 1
I20260809 03:34:43.868330 19512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135ea63f72ee40c8b25ad7596f42b387 in term 1.
I20260809 03:34:43.868587 19386 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135ea63f72ee40c8b25ad7596f42b387 in term 1.
I20260809 03:34:43.868850 19221 leader_election.cc:302] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135ea63f72ee40c8b25ad7596f42b387, e118d8dabf114541af326f7215571595; no voters:
W20260809 03:34:43.871546 19601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:43.871620 19601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 03:34:43.872260 19225 tablet.cc:2369] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 03:34:43.872432 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.872548 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.872610 19586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 03:34:43.872664 19586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [term 1 LEADER]: Becoming Leader. State: Replica: 135ea63f72ee40c8b25ad7596f42b387, State: Running, Role: LEADER
I20260809 03:34:43.872953 19586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } }
I20260809 03:34:43.870246 19188 master_runner.cc:186] Error getting master registration for 127.15.222.125:43281: OK, instance_id { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" instance_seqno: 1786246483591134 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 03:34:43.874945 19188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 03:34:43.873930 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.875038 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.875324 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ea63f72ee40c8b25ad7596f42b387. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.875443 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: This master's current role is: LEADER
W20260809 03:34:43.877607 19629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 03:34:43.877751 19629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 03:34:43.884629 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.874038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.15.222.124:35595)
0809 03:34:43.874198 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.874203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.875853 (+ 1650us) server_negotiation.cc:407] Connection header received
0809 03:34:43.876067 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.876070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.876120 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.876171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.877040 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.877805 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.878869 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.879194 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.879284 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.881023 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.881027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.881029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.881057 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.882730 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.883235 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.883238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.883240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.883335 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.883652 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.883656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.883658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.883861 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.883987 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.884339 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.884472 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 03:34:43.887290 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.878599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.15.222.126:42075)
0809 03:34:43.878699 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.878703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.879134 (+ 431us) server_negotiation.cc:407] Connection header received
0809 03:34:43.879300 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.879303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.879345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.879393 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.880298 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.881008 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.882092 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.882408 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.882484 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.883979 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.883983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.883985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.884010 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.885514 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.886002 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.886005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.886007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.886093 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.886376 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.886379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.886381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.886584 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.886679 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.887043 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.887149 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 03:34:43.893699 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:43.879363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.15.222.125:43281)
0809 03:34:43.879487 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:43.879491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:43.886119 (+ 6628us) server_negotiation.cc:407] Connection header received
0809 03:34:43.886277 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:43.886280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:43.886327 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:43.886375 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:43.887053 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.887730 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.888469 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:43.888681 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:43.888763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:43.890256 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 03:34:43.890260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 03:34:43.890263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 03:34:43.890288 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 03:34:43.891876 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.892351 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.892355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.892357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.892455 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 03:34:43.892754 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 03:34:43.892757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 03:34:43.892759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 03:34:43.892962 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 03:34:43.893069 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:43.893436 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 03:34:43.893550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 03:34:43.897982 19636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 03:34:43.899051 19636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 03:34:43.908710 19386 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [term 1 FOLLOWER]: Refusing update from remote peer 135ea63f72ee40c8b25ad7596f42b387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:43.908756 19512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [term 1 FOLLOWER]: Refusing update from remote peer 135ea63f72ee40c8b25ad7596f42b387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 03:34:43.909319 19586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:43.909701 19624 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 03:34:43.912935 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ea63f72ee40c8b25ad7596f42b387. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.913049 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.913128 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ea63f72ee40c8b25ad7596f42b387. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.913230 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.926317 19644 mvcc.cc:205] Tried to move back new op lower bound from 7316465598073163776 to 7316465597949558784. Current Snapshot: MvccSnapshot[applied={T|T < 7316465598073163776}]
I20260809 03:34:43.926832 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.926941 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:43.928068 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.928167 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.928577 19648 mvcc.cc:205] Tried to move back new op lower bound from 7316465598073163776 to 7316465597949558784. Current Snapshot: MvccSnapshot[applied={T|T < 7316465598073163776}]
I20260809 03:34:43.937191 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.937301 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387 [sys.catalog]: This master's current role is: LEADER
I20260809 03:34:43.938532 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "135ea63f72ee40c8b25ad7596f42b387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135ea63f72ee40c8b25ad7596f42b387" member_type: VOTER last_known_addr { host: "127.15.222.126" port: 42075 } } peers { permanent_uuid: "93a12e8ba6874e64a559d48df27dfc77" member_type: VOTER last_known_addr { host: "127.15.222.125" port: 43281 } } peers { permanent_uuid: "e118d8dabf114541af326f7215571595" member_type: VOTER last_known_addr { host: "127.15.222.124" port: 35595 } } }
I20260809 03:34:43.938642 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 03:34:43.939345 19646 mvcc.cc:205] Tried to move back new op lower bound from 7316465598073163776 to 7316465597949558784. Current Snapshot: MvccSnapshot[applied={T|T < 7316465598073163776}]
I20260809 03:34:43.942288 19636 catalog_manager.cc:1347] Generated new cluster ID: 2de230d00a4143399362fcbaa8f2ab5e
I20260809 03:34:43.942344 19636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 03:34:43.964051 19636 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 03:34:43.965407 19636 catalog_manager.cc:1504] Loading token signing keys...
I20260809 03:34:43.977991 19636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135ea63f72ee40c8b25ad7596f42b387: Generated new TSK 0
I20260809 03:34:43.978995 19636 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 03:34:44.032887 19561 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.029682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.15.222.124:35595)
0809 03:34:44.029858 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.029863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.031786 (+ 1923us) server_negotiation.cc:407] Connection header received
0809 03:34:44.031987 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.031991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.032044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.032094 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.032745 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57576: BlockingRecv error: recv got EOF from 127.0.0.1:57576 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 03:34:44.034557 19311 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.030443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.15.222.126:42075)
0809 03:34:44.030552 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.030556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.033561 (+ 3005us) server_negotiation.cc:407] Connection header received
0809 03:34:44.033764 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.033768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.033825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.033880 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.034437 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50860: BlockingRecv error: recv got EOF from 127.0.0.1:50860 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 03:34:44.036605 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.031187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.15.222.125:43281)
0809 03:34:44.031302 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.031306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.034017 (+ 2711us) server_negotiation.cc:407] Connection header received
0809 03:34:44.035457 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.035461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.035544 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.035629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.036422 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57296: BlockingRecv error: recv got EOF from 127.0.0.1:57296 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 03:34:44.045879 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.040436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.15.222.125:43281)
0809 03:34:44.040535 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.040539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.044925 (+ 4386us) server_negotiation.cc:407] Connection header received
0809 03:34:44.045139 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.045142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.045214 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.045264 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.045780 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57298: BlockingRecv error: recv got EOF from 127.0.0.1:57298 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 03:34:44.051195 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.049859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.15.222.125:43281)
0809 03:34:44.049972 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.049976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.050282 (+ 306us) server_negotiation.cc:407] Connection header received
0809 03:34:44.050544 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.050547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.050618 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.050687 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.051101 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57302: BlockingRecv error: recv got EOF from 127.0.0.1:57302 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 03:34:44.058154 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.056533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.15.222.125:43281)
0809 03:34:44.056636 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.056640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.057175 (+ 535us) server_negotiation.cc:407] Connection header received
0809 03:34:44.057368 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.057372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.057436 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.057490 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.058047 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57312: BlockingRecv error: recv got EOF from 127.0.0.1:57312 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 03:34:44.083226 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.074244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.15.222.125:43281)
0809 03:34:44.074415 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.074419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.082073 (+ 7654us) server_negotiation.cc:407] Connection header received
0809 03:34:44.082324 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.082328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.082391 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.082449 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.083093 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57324: BlockingRecv error: recv got EOF from 127.0.0.1:57324 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260809 03:34:44.094568 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.093379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.15.222.125:43281)
0809 03:34:44.093498 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.093503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.093754 (+ 251us) server_negotiation.cc:407] Connection header received
0809 03:34:44.093896 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.093900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.093948 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.094015 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.094448 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57336: BlockingRecv error: recv got EOF from 127.0.0.1:57336 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 03:34:44.113610 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.112609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.15.222.125:43281)
0809 03:34:44.112714 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.112717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.112896 (+ 179us) server_negotiation.cc:407] Connection header received
0809 03:34:44.113021 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.113023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.113069 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.113145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.113508 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57342: BlockingRecv error: recv got EOF from 127.0.0.1:57342 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 03:34:44.148703 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.147607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.15.222.125:43281)
0809 03:34:44.147754 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.147759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.147866 (+ 107us) server_negotiation.cc:407] Connection header received
0809 03:34:44.148043 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.148046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.148093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.148161 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.148604 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57354: BlockingRecv error: recv got EOF from 127.0.0.1:57354 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 03:34:44.215906 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.214778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.15.222.125:43281)
0809 03:34:44.214934 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.214939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.215039 (+ 100us) server_negotiation.cc:407] Connection header received
0809 03:34:44.215203 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.215206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.215253 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.215319 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.215774 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57366: BlockingRecv error: recv got EOF from 127.0.0.1:57366 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 03:34:44.346928 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.345808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.15.222.125:43281)
0809 03:34:44.345948 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.345952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.346062 (+ 110us) server_negotiation.cc:407] Connection header received
0809 03:34:44.346235 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.346237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.346286 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.346351 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.346803 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57370: BlockingRecv error: recv got EOF from 127.0.0.1:57370 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 03:34:44.606029 19440 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 03:34:44.605038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.15.222.125:43281)
0809 03:34:44.605157 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:44.605160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:44.605254 (+ 94us) server_negotiation.cc:407] Connection header received
0809 03:34:44.605373 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:44.605376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:44.605425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:44.605494 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 03:34:44.605921 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57386: BlockingRecv error: recv got EOF from 127.0.0.1:57386 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 03:34:44.859722 19579 catalog_manager.cc:1259] Loaded cluster ID: 2de230d00a4143399362fcbaa8f2ab5e
I20260809 03:34:44.859803 19579 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: loading cluster ID for follower catalog manager: success
I20260809 03:34:44.861604 19579 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: acquiring CA information for follower catalog manager: success
I20260809 03:34:44.862846 19579 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e118d8dabf114541af326f7215571595: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:34:44.879501 19629 catalog_manager.cc:1259] Loaded cluster ID: 2de230d00a4143399362fcbaa8f2ab5e
I20260809 03:34:44.879576 19629 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: loading cluster ID for follower catalog manager: success
I20260809 03:34:44.881207 19629 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: acquiring CA information for follower catalog manager: success
I20260809 03:34:44.882403 19629 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93a12e8ba6874e64a559d48df27dfc77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 03:34:45.123971 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:45.120140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.15.222.125:43281)
0809 03:34:45.120383 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:45.120388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:45.120404 (+ 16us) server_negotiation.cc:407] Connection header received
0809 03:34:45.120553 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:45.120556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:45.120686 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:45.120782 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:45.121080 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.121611 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.122524 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.122745 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.122836 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:45.123141 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:45.123590 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:45.123698 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:45.123765 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 03:34:45.123821 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260809 03:34:45.129769 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:45.126724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.15.222.124:35595)
0809 03:34:45.126961 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:45.126964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:45.126977 (+ 13us) server_negotiation.cc:407] Connection header received
0809 03:34:45.127052 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:45.127054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:45.127166 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:45.127256 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:45.127402 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.127880 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.128591 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.128791 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.128852 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:45.129065 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:45.129448 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:45.129531 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:45.129558 (+ 27us) server_negotiation.cc:299] Negotiation successful
0809 03:34:45.129602 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260809 03:34:45.136068 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:45.132373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.15.222.124:35595)
0809 03:34:45.132575 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:45.132579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:45.132986 (+ 407us) server_negotiation.cc:407] Connection header received
0809 03:34:45.133145 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:45.133148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:45.133195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:45.133252 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:45.133491 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.134205 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.135012 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.135252 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.135333 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:45.135627 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:45.135702 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:45.135771 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:45.135909 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 03:34:45.135956 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260809 03:34:45.136117 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:45.132313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.15.222.125:43281)
0809 03:34:45.132500 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:45.132504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:45.132729 (+ 225us) server_negotiation.cc:407] Connection header received
0809 03:34:45.132854 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:45.132858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:45.132906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:45.132954 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:45.133277 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.134030 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.135062 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.135371 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.135473 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:45.135689 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:45.135759 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:45.135824 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:45.135946 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 03:34:45.135997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260809 03:34:45.136423 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 03:34:45.132287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.15.222.126:42075)
0809 03:34:45.132529 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 03:34:45.132533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 03:34:45.132548 (+ 15us) server_negotiation.cc:407] Connection header received
0809 03:34:45.132719 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 03:34:45.132722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 03:34:45.132875 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 03:34:45.132990 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 03:34:45.133197 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.133838 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.134938 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 03:34:45.135265 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 03:34:45.135619 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 03:34:45.135742 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 03:34:45.136049 (+ 307us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 03:34:45.136132 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 03:34:45.136223 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 03:34:45.136282 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260809 03:34:45.139298 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19188
I20260809 03:34:45.151702 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19315
I20260809 03:34:45.161979 16249 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbUfAuu/build/debug/bin/kudu with pid 19444
2026-08-09T03:34:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3196 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6350 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284587 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST